MFactor Documentation
Table of Contents
- 1. Introduction
- 2. TODO Supplied Libraries
- 3. Host Compiler
- 4. VM Implementation
- 4.1. Definitions and Datatypes
- 4.2. Work Memory
- 4.3. Special Variables
- 4.3.1. Special Variables Definition
- <IP_ADDRESS>.
MP
: The Memory Write Pointer - <IP_ADDRESS>.
HANDLER
: Current Exception handling frame - <IP_ADDRESS>.
DEBUG_LEVEL
: VM Debug Verbosity - <IP_ADDRESS>.
RESTART
: Restart address (deprecated) - <IP_ADDRESS>.
STEP_HANDLER
: Single Stepping Handler (not in use) - <IP_ADDRESS>.
BASE
: Current Segment base address - <IP_ADDRESS>.
OUTPUT_STREAM
: Output Stream descriptor
- <IP_ADDRESS>.
- 4.3.2. Special Variables Initialization
- 4.3.1. Special Variables Definition
- 4.4. Helper Funtions
- 4.5. Used Subroutines (defined as Macros)
- 4.6. Main Interpreter function
interpreter()
- 4.6.1. Internal State Definition
- 4.6.2. State initialization
- 4.6.3. Primitive Instructions
- <IP_ADDRESS>. Basic stack manipulation
- <IP_ADDRESS>. Arithmetic Instructions
- <IP_ADDRESS>. Constants
- <IP_ADDRESS>. Access to Internal Constants
- <IP_ADDRESS>. Access to Internal State
- <IP_ADDRESS>. Literal Instructions
- <IP_ADDRESS>. IO Primitives
- <IP_ADDRESS>. Call/Return Instructions
- <IP_ADDRESS>. Conditional
- <IP_ADDRESS>. Memory Access
- <IP_ADDRESS>. Foreign Function Execution
- <IP_ADDRESS>. (As of Yet) Undocumented instructions
- 5. Appendix: Complete Sources
1 Introduction
MFactor is a (partial) implementation of the factor language for embedded systems. It consists of two parts:
- A host compiler, written in Ruby, which is used with the Rake build system to produce a binary image (see Host Compiler).
- A c implementation of a bytecode vm which is able to execute such an image. (see VM Implementation)
It is intended to be used with a embedded C development environment, where its sources are compiled together with the host application.
1.1 Usage Concept
The general idea: First compile .mfactor source files to bytecode image, then compile the provided c source files together with the host application's source files into the final binary.
A set of mfactor source files is translated by the host compiler (called through rake task
mfactor
) into a bytecode image. This bytecode image is then successively executed by
interpeter()
, which is the actual VM implementation.
In order to incorporate this into a project, several rake tasks are supplied in tasks/mfactor.rake
- mfactor: the main task, which, when called, generates above code
- mftest[word name]: helper task, outputs a graphical representation of a specific word in the
generated
subdir.
1.2 Contents
Description of files and directories.
Gemfile[.lock]
- Ruby packages version, see ruby gems documentation
Rakefile
- for mfactor development only
instructionset.yml
- this file defines the primitive instructions for the VM
Implementation. Note that the host compiler gets its list of supported instructions
from
src/mfactor/primitives.mfactor
.
1.2.1 html
contains generated documentation
1.2.2 lib
Ruby source code for the Host Compiler.
1.2.3 org
Source of this file
1.2.4 src
Contains the target-independend C source, some linker files and the supplied MFActor libraries (see 2)
interpreter.[ch]
- source of the VM Implementation
reader.c
- simple tokenizer implementation
seq_headers.h
- definitions for inline data headers(e.g. used by
litc
in <IP_ADDRESS>
The following headers have target-dependent implementations in target/
reset_system.h
- toggle system/interpreter reset
runtime.h
- platform-specifics for measuring run times
<IP_ADDRESS> mfactor
Supplied MFactor libraries (see 2).
1.2.5 tasks
mfactor.rake
: The file that defines the mfactor
rake task that is used to integrate
the VM into the host application (see 1.3)
1.2.6 test
Ruby unit tests
1.3 Configuring the Rake Task
The rake file is supposed to be included with rake's import
facility in the host
application's own Rakefile.
Currently, it is configured via setting global variables in the including rake file. The following must be set:
MFACTOR_SRC_DIR
: Determines the directory where the application's mfactor sources can be found. This should point to the application's mfactor source directory.GENERATOR
: which bytecode backend to use, currently, onlyCortex
is supportedSTART_WORD
: word, which will be executed when starting the vmMFACTOR_ROOT_VOCAB
: The root vocabulary, which is first loaded and loads all dependencies. Must be found inMFACTOR_SRC_DIR
.MFACTOR_DEPENDING_OBJECTS
: To cause Rake to recompile the whole image, this should be set to the object files in the application which depend on the generated output. This usually means all files that use any of the words defined inMFACTOR_C_WORDS
, as well as the object file that is built fromsrc/interpreter.c
The following are for advanced functionality:
MFACTOR_C_WORDS
: Hashtable for exposing word locations in bytecode to C. Format:{ "mfactor-word-name" => "NAME_OF_CPP_DEFINE" }
This can be used to generate preprocessor symbols, which can be passed directly to
interpreter()
, for example.MFACTOR_FF
: path to file containing interface definitions to c code (see 1.5)MFACTOR_IMAGE_SECTION
: a string describing the linker section that the bytecode-image will pe placed intoMFACTOR_DICT_SECTION
: same thing for the generated dictionary
1.4 Generated Output Files
In the application's main directory, a directory generated
is created during the build
process, where several files are generated:
bytecode.c
- main output from the ByteCodeImage class.
bytecode.h
- consists of defines that
interpreter()
and maybe the user application needs to know about image.dump
- after translation, the ByteCodeImage is written here using Marshal. It
can be reloaded later in the build process with
ByteCodeImage.deserialize
to extract information about the image.
1.5 Making Existing C Functions Available
Special global variable MFACTOR_FF
can be set to a yaml file for
"importing" existing c functions into the interpreter's namespace (Foreign function interface).
entries are in the form of:
<c_name>: name: "<mfactor-name>" call: <callspec> # following optional line is for void functions, for functions returning a value, don't use result: discard
where <callspec>
describes the function's arguments, e.g. "iis" for a function like fn(int, int, int16)
.
 The language is concatenative, and uses generally postfix syntax, e.g. the arguments are written before the "function call". This provides a good introduction to the concepts of concatenative languages.
Terminology:
- Word
- the basic user-defined abstraction, like functions in most languages.
- Parsing Word
- defined by
SYNTAX:
, these take control of the parser if used interactively, and are supported indepently of that by the byte-code compiler. - Vocabulary
- comparable to C's compilation unit with interface definition in a header file. Contains a number of word definitions.
- Data Stack
- or Parameter Stack, holds values that are manipulated by words.
- Return Stack
- keeps track of program execution when calling into words and returning. In contrast to Forth, not directly accessible by user code.
- Retain Stack
- second stack for data, which is used to set aside stack elements that are needed for later. Used internally by all preserving combinators.
- Quotation
- block of code enclosed in square brackets (
[]
), that is pushed onto the stack as-is, without evaluation. - Combinator
word that takes quotations as input. The conditional word
if
takes a condition and 2 quotations as input:condition-value [ true-quotation ] [ false-quotation ] if
- Primitives
- are the low-level instructions that are executed by the VM. All words are defined in terms of other words, some of them primitives. The binary representation of a word, however, consists entirely of primitives.
- Image
- is a (compiled) array of primitives, that constitutes the binary representation of the mfactor source program
- Dictionary
- generated table that holds information on word definitions, most prominently the address of the words's bytecode in the image
1.7 Differences to Factor
Mfactor implements a subset of factor's functionality useful for embedded systems programming. Major differences:
- no namestack, so no dynamic variables (may change, but high performance impact for embedded systems)
- vocabulary search is simplified, vocabulary
foo
would be found in<MFACTOR_SRC_DIR>/foo.mfactor
, vocabularyfoo.bar
would be found in<MFACTOR_SRC_DIR>/foo.bar.mfactor
. - same-named words in different vocabularies not supported (yet)
- interactive support only for byte-array and integer-array sequences:
B{ }
,I{ }
, at compile time, arbitrary sequences can be used. - host compiler only supports a subset of parsing words:
:
,SYNTAX:
,{ }
,B{ }
,I{ }
, `SYMBOLS:`,USING:
,IN
case
is implemented as a syntax word, so no interactive support- words beginning with underscore(
_
) are not stored in the dictionary. This is intended for helper functions, to prevent them from consuming dictionary space. - no continuations, no quotation compositions(yet)
- simplified exception handling with `catch` and `throw`
- no Garbage Collector. Since interactive usage is meant to be for developing, allocated
objects by the reader are not released automatically. Any data used in normal code is
statically compiled into the image, so no runtime allocation is performed unless
specified explicitely using sequence constructors
<byte-array>
and<integer-array>
- no object system, so the type of stack items can generally not be determined (with exceptions. If it is known to point to a sequence or other complex data structure, information about the element types can be extracted)
Practical Consequences: Most of the omissions are due to resource constraints in embedded systems, where runtime allocation is generally not desirable. The missing runtime type support has the following main consequences:
When printing an object, it must be known what it is:
.
,.x
,.b
shows the raw value(decimal, hex, binary) of an item, if it is a number, that is its value, if it is a sequence, it is the address of it's count byteba.x
,ba.b
prints the elements of a byte-array (hex, binary). If top of stack is not a byte array, garbage is output.- generally (e.g. for integer arrays) normal iteration also works:
some-integer-array [ . ] each
Not all sequence operations are type-independent (yet). Support matrix:
| factor | mfactor general | mfactor for byte arrays | mfactor for integer arrays |
|---|---|---|---|
| nth | nth | ba-nth | ia-nth |
| set-nth | N/A | ba-set-nth | ia-set-nth |
| each | each | ba-each | ia-each |
| change-nth | N/A | ba-change-nth | N/A |
| find | N/A | ba-find | N/A |
| reduce | reduce | ba-reduce | N/A |
| each-index | each-index | N/A | N/A |
2 TODO Supplied Libraries
A number of mfactor source files are supplied together with the VM, to provide the (mainly) Factor-compatible functionality. These are to be documented here.
2.1 TODO Listener
Add control flow example using the listener.
3 Host Compiler
3.1 Concept
The host compiler scans MFACTOR_SRC_DIR
for the vocabulary specified with
MFACTOR_ROOT_VOCAB
, and loads it. When loading a vocabulary, it searches it's own
mfactor sources and the files in MFACTOR_SRC_DIR
for a file named
<vocab-name>.mfactor
. It recursively loads any formerly unloaded vocabularies in the
process. Loading a vocabulary involves following ruby classes:
The Parser reads the source code, checking for syntax errors and generating an internal
representation of the code. Once all the words have been loaded, the internal Image
object is turned into a ByteCodeImage
object, which is the main output product.
3.2 TODO Parser details
3.3 Bytecode Generator Output
Per default, the output of the rake task is place in subdirectory generated
of the
including project. This folder contains several files:
inst_enum.h
- enum which contains all instruction mnemonics. These are used in the image generated in =bytecode.c"mfactor_words.h
- contains all exported words, which can then be referenced from C context.bytecode.c
- contains the actual byte code image, dictionary and hash tablebytecode.h
- some constants which are generated during byte code compilation and used in the VM implementationimage.dump
- serialized MFactor::ByteCodeImage Object, can be loaded withMFactor::ByteCodeImage::deserialize
. This basically exposes all possible internals about the compiled image. Used to extract information after compilation, when themfactor
task has finished.
3.4 Invocation
If MFACTOR_DEPENDING_OBJECT
was set correctly in the host application's Rakefile, then a
rebuild of the application automatically triggers a recompile of the bytecode image.
3.5 TODO Compilation example
3.6 Details
3.6.1 Fried Quotations(WIP)
Fried Quotations (and their basic versions curry
and compose
) are the equivalent of
closures. Creating a closure always means allocating space somewhere. If the closure
creation is inlined, that space can be allocated by the compiler. Therefore, fried
quotations are currently only supported on the host compiler, and all quotations
containing them are automatically inlined. Also, if looping code is used to return
multiple results of fried quotation definitions, note that they will be equal to the last
invocation, since no allocation is performed at compile time. In theory, a runtime check
could be implemented which signals a runtime error when a quotation is being fried more
than once.
To describe the transformation, consider the following example:
! for each i where i runs from 0 below n, x is added to it before applying quot to it : each-int-added ( n x quot ( i -- ) -- ) '[ _ + @ ] each-integer; inline
Let's define a word that makes use of this word:
! print values that are passed to the quotation in each-int-added, ! the value added to each element is x, it is called 10 times : test ( x -- ) 10 swap [ . ] each-int-added ;
Conceptually, the following series of transformations is appplied internally when defining test
:
test ( n -- ) 10 swap [ . ] '[ _ + @ ] each-integer
test ( n -- ) 10 swap [ . ] '[ _ + _ call ] each-integer
test ( n -- ) 10 swap [ . ] 0x0002 setmem 0x0001 setmem [ (0x0001) + (0x0002) call ] each-integer
The locations sites where the "curried-in" data is used are made explicit, and before the
quotation is passed to each-integer
, code is generated that sets the actual memory
locations to their values at runtime. This is done even when the values are constant and
known at compile time.
Since it is effectively rewriting the quotation at runtime, this incurs a small performance hit. That also illustrates why these words have to be inlined. For every use, the quotation must essentially be copied if the rewritten items are to not interfere with different invocations.
the @
specifier actually does no splicing, but simply calls the original quotation.
This must be kept in mind if the spliced quotation is mutable in any way.
If code is stored in non-writable memory, another indirection is needed. The placeholders are not substituted directly, but filled at runtime by a read to a known writable location:
test ( n -- ) 10 swap 0x0001 setmem [ . ] 0x0002 setmem '[ 0x0001 getmem + 0x0002 getmem call ] each-integer
This way, no actual code rewrite is performed, but the item use costs another indirection. Effectively, this allocates "variables" in a data segment, which the compiler assumes to be writable at runtime.
Note that both versions introduce the overhead of copying the code of the original definition containing the fry at every call site!
An obvious optimization here is to separate the last part of the definition, after the fried quotation, into an own word. This is very probably almost always the case anyways, as the example definition above illustrates that typically the quotation is passed to another combinator. If that one contains fried quotations, it will obviously further increase the copied code overhead.
4 VM Implementation
The VM is a stack machine with three stacks, a data stack (or parameter stack, pstack), return stack and a retain stack.
The data stack is used for argument passing, while the return stack is used to save the program counter during sub-routines (and store some debug information). The retain stack is only used for putting stack items out of the way temporarily.
A program counter points to successively executed primitive instructions.
Instruction Set The Instruction Set for the VM is defined in instructionset.yml. For a description of the instructions skip to 4.6.3.
The following Section describes all definitions that end up in the header file src/interpreter.h, the rest describes the implementation in src/interpreter.c
4.1 Definitions and Datatypes
In src/interpreter.h are relevant data type definitions and preprocessor macro defaults. These are supposed to be overriden to configure the compiled runtime (see ???).
4.1.1 Typedefs
<IP_ADDRESS> Scalar Types
The type of actual primitive instructions which are loaded and evaluated, and from which byte code images are constructed. (see ???) Size: 1 byte
typedef unsigned char inst;
Targets of normal jumps and calls. 16 bit size, so if an image is bigger than 64K, these are not sufficient (see ???call instructions) Size: 2 bytes
typedef unsigned short short_jump_target; /* relative jumps in 64k on 32 bit */
Targets of long jumps. Use full 32 Bit address space. Used for calls to addresses on stack. Size: 32 Bit
typedef uintptr_t jump_target; /* long absolute jump */
Type of data actually manipulated on the stack. Size: 32 Bit
typedef uintptr_t cell; /* memory cell must at least hold pointer */
<IP_ADDRESS> Dictionary entries
Type of entries in the dictionary. These map names to addresses, and hold the flags that mark a word inline, recursive, or a parsing word.
typedef struct dict_entry { inst * address; /* pointer into memory */ unsigned char flags; /* may include other flags later (inline, recursive, etc) */ unsigned char name_header; /* should always be zero */ unsigned char name_length; char name[]; } __attribute__((packed)) dict_entry;
Later, the dictionary entries are collected into the dictionary generated in
bytecode.c
. Note that is not really possible to access entries in that array by index,
since the entries are have the last field as variable-length data on initialization.
4.1.2 Preprocessor Macros
The following all indicate the size of the different components, in cell
units.
These are defaults can be changed according to application requirements.
/* data memory (affects non-transient data) in cells*/ #ifndef VM_MEM #define VM_MEM 256 #endif /* dictionary size (affects number of named items)*/ #ifndef VM_DICT #define VM_DICT 512 #endif /* parameter stack size (affects transient data)*/ #ifndef VM_PSTACK #define VM_PSTACK 64 #endif /* return stack size (affects nesting of functions)*/ #ifndef VM_RETURNSTACK #define VM_RETURNSTACK 64 #endif /* retain stack size (affects maximum amount of postponing data use) */ #ifndef VM_RETAINSTACK #define VM_RETAINSTACK 32 #endif
Another macro can be preset or computed: INSTBASE
. This is used to distinguish
primitive instructions from quotations, when executing words on the stack. Primitives
cannot be interpreted as memory addresses, since these would point into invalid memory.
On Cortex-M, all memory addresses higher than 0x80… are not accessable, and can be used for primitive instructions.
#ifndef INSTBASE #if (__linux && __LP64__) #define INSTBASE 0x80U #elif (CORTEX_M) #define INSTBASE 0xA0U #else #error "don't know instruction code base for architecure!" #endif #endif
This influences the opcodes generated in inst_enum.h
during building.
A cell
-sized version for comparison to data values:
#define INSTBASE_CELL ((cell)INSTBASE<<(8*(sizeof(inst *)-sizeof(inst))))
4.1.3 Defines for the generated data in bytecode.c
extern inst image[]; extern dict_entry dict[VM_DICT]; extern uint16_t dict_hash_index[]; extern cell FF_Table[];
image
holds the actual firmware image, dict
is the dictioary, dict_hash_index
creates a hash table for fast lookup of words (see ???). dict
is declared with size
because in the lookup function the sizeof()
operator is used for bounds-checking.
4.1.4 Internal Error Codes
These are returned from interpreter() in case of a unrecoverable internal error (e.g. stack underflow).
/* these values can be returned from interpreter() */ #define INTERNAL_ERROR_PSTACK_UFLOW -1 #define INTERNAL_ERROR_PSTACK_OFLOW -2 #define INTERNAL_ERROR_RSTACK_UFLOW -3 #define INTERNAL_ERROR_RSTACK_OFLOW -4 #define INTERNAL_ERROR_RTSTACK_UFLOW -5 #define INTERNAL_ERROR_RTSTACK_OFLOW -6 #define INTERNAL_ERROR_INVALID_OPCODE -7 #define INTERNAL_ERROR_MEM_FAULT -8 #define INTERNAL_ERROR_UNKNOWN_FF -9 #define INTERNAL_ERROR_TOKEN_READ -10 #define INTERNAL_ERROR_GENERAL -11 #define INTERNAL_ERROR_UNKNOWN_SPECIAL -12
4.1.5 Main VM Function Prototype
This is the prototype for the function that is supposed to be executed from the application program. The only argument is the offset of the first in the bytecode image to be executed. The return value indicates the status after a bytecode-program has been executed. If it is non-zero, an internal error happened.
int interpreter(short_jump_target);
4.2 Work Memory
For allocation of interactively-generated input sequences, such as strings, byte arrays and quotations, internal memory is reserved. This is the default target area for the compilation primitives (TODO: link).
/* main memory to work with */ static cell memory[VM_MEM];
4.3 Special Variables
4.3.1 Special Variables Definition
The internal state of the interpreter is exposed to the application by predefining an array of special variables. These are needed for all instructions that depend on state that must be influenced by the application.
First define an array that holds these Variables:
#define _NumSpecials 10 static const unsigned char NumSpecials = _NumSpecials; static cell special_vars[_NumSpecials];
The list of currently implemented specials:
/* currently implemented special variables 0: MP - memory write pointer 1: HANDLER - handler frame location in r(etain) stack (dynamic chain for catch frames) 2: DEBUG_LEVEL - 0 to turn off, increasing will produce more verbose debug output 3: RESTART - word where to restart when hard error occured 4: STEP_HANDLER - address of handler which can be used for single stepping 5: BASE - address of current 64k segment base 6: OUTPUT_STREAM: 1: stdout, 2: stderr, 3: null */
Not all of them are actually in use, the ones that are, are defined as macros so that the
special_vars
array needs not be acccessed directly in the following source code:
<IP_ADDRESS> MP
: The Memory Write Pointer
This holds a pointer to the memory location that is accessed by the compilation primitives (TOOD: link)
#define MP special_vars[0]
<IP_ADDRESS> HANDLER
: Current Exception handling frame
This is used by the application to store information about the current exception handler. Note that the application does not currently manipulate or access this state, so it is basically a specialized global variable. TODO: reference the exception handling lib
#define HANDLER special_vars[1]
<IP_ADDRESS> DEBUG_LEVEL
: VM Debug Verbosity
Used to activate debug output for VM internals. Only used for VM development and debugging.
#define DEBUG_LEVEL special_vars[2]
<IP_ADDRESS> RESTART
: Restart address (deprecated)
This used to hold an address that was jumped to in case of internal errors. This has been
replaced by returning from interpreter()
with the internal error code, and the host
application deciding what to do then
<IP_ADDRESS> STEP_HANDLER
: Single Stepping Handler (not in use)
This shall hold an address which is called on every step of the currently executed bytecode image. Meant for implementing single step debugging, where the actual handler would be written as mfactor source.
<IP_ADDRESS> BASE
: Current Segment base address
This hold the address that is added to all internal calls (TODO: link to relevant call
primitives). Usually it holds the address of the image
-variable, that is generated in
generated/bytecode.c
. This allows the image to be relocatable, so that the actuall call
target (16 bit value) is computed by adding it to the value of BASE
.
#define BASE special_vars[5]
<IP_ADDRESS> OUTPUT_STREAM
: Output Stream descriptor
The value of this variable is used everytime something should be output. It determines
which File
pointer is passed to the standard c output functions in current_fd()
(see
4.4.1)
#define OUTPUT_STREAM special_vars[6]
The possible values are defined as:
/* known stream descriptors for OUTPUT_STREAM */ #define STDOUT 1 #define STDERR 2 #define NULLOUT 3
4.3.2 Special Variables Initialization
This helper is called at the beginning of interpreter()
to initialize some special
variables to values then known.
static void init_specials() { HANDLER = 0; MP = (cell)memory; /* start of user memory */ BASE = (cell)ℑ /* start of bytecode segment */ OUTPUT_STREAM = STDOUT; /* output to standard output per default */ }
4.4 Helper Funtions
4.4.1 Output Handling
One file-local variable that holds the current Output stream for io operations:
static FILE * Ostream; /* used by reporting functions, so they can temporarily print to different file descriptor */ /* get the current stdio FILE from the special variable, or NULL if unknown or muted by * choosing NULLOUT */ static FILE * current_fd(void) { if (OUTPUT_STREAM == 2) return stderr; else if (OUTPUT_STREAM == 1) return stdout; else return NULL; }
4.4.2 Return Stack Entry Format
The return stack, which holds the links of the dynamic chain, has entries in the following format:
typedef struct return_entry { inst * return_address; inst * current_call; } return_entry;
Technically, for execution, only the return_address
portion is interesting. But for
debugging purposes and backtraces it is useful to record the starting address of the
current quotation. This is done with the current_call
field.
4.4.3 Word Lookup: Dictionary support
There is built-in support for work lookup in the dictionary (see TODO: link to dictionary explanation), mainly for speed reasons. During compilation, a simple hash table was created by the ruby translator (TODO: link). This code is the runtime access portion. It uses a very simple hash function to look up a starting index that can be used for looking up a word with a linear search from there.
uint32_t lookup_ht_entry(uint8_t length, char* name) { uint32_t hash = 5381; for (int i = 0; i < length; i++) { hash = hash * 33 + name[i]; } return (cell)dict+dict_hash_index[hash%256]; }
For debugging, there is a (slow) reverse lookup, that gets the word name from its execution address:
/* get the name of the word, only for debugging */ /* probably fails for non-null-terminated strings */ static char* find_by_address( inst * word) { static char notfound[] = "(internal or private)"; for (char * ptr=(char*)dict; (ptr < ((char*)dict+sizeof(dict)))&&(((dict_entry*)ptr)->name_length > 0); ptr += (((dict_entry*)ptr)->name_length + 3*sizeof(unsigned char) + sizeof(void*))) { dict_entry *dptr = (dict_entry*)ptr; if (dptr->address == word) return dptr->name; } return notfound; }
4.4.4 Parsing Numbers
Whenever a piece of input is unknown, it is tried to parse as a number before giving up.
This uses libc's sscanf()
, and is exposed via the parsenum
primitive, which uses this
function.
static bool parse_number(char *str, cell * number){ int num; unsigned int read = sscanf(str,"%i",&num); if (read == 1) { *number = (cell)num; return true; } else { return false; } }
4.4.5 Debug output
For the st
primitive and in case of errors, the stack contents are sometimes printed to
stderr
. This is done with the following file-local functions:
/* display data stack entries */ static void printstack(cell * sp, cell * stack) { fprintf(Ostream, "stack:"); for(cell* ptr = stack;ptr < sp;ptr++) { fprintf(Ostream, " %#lx",*ptr); } fprintf(Ostream, "\n"); } /* display return stack entries */ static void print_return_stack(return_entry * sp, return_entry * stack, inst * base) { fprintf(Ostream, "stack:"); for(return_entry* ptr = sp-1;ptr >= stack;ptr--) { fprintf(Ostream, " {%#lx->%#lx}",((uintptr_t)ptr->current_call-(uintptr_t)base), ((uintptr_t)ptr->return_address)-(uintptr_t) base); } fprintf(Ostream, "\n"); } /* print a backtrace of the return stack */ static void backtrace(return_entry * sp, return_entry * stack, inst * base, inst * pc) { fprintf(Ostream, "backtrace @ %#lx:\n",(uintptr_t)(pc-base)); for(return_entry* ptr = sp-1;ptr >= stack;ptr--) { char *current_name = find_by_address(ptr->current_call); fprintf(Ostream, "%#lx %s\n",(uintptr_t)(ptr->current_call - base),current_name); } }
Helper for printing errors:
static void print_error(char * str) { fprintf(stderr, "error: "); fprintf(stderr, str); fprintf(stderr, "\n"); }
4.5 Used Subroutines (defined as Macros)
There are several subroutines that are repeatedly used in interpreter()
, which operate
on the static data that is only defined there. Because of that, they are implemented as
preprocessor macros.
4.5.1 Backtrace printing
#define BACKTRACE() do { \ FILE * old_out = Ostream; \ Ostream = stderr; \ printstack(psp,pstack); \ printstack(retainsp,retainstack); \ backtrace(returnsp,returnstack,(inst *)BASE,pc); \ Ostream = old_out; \ } while (0)
4.5.2 Errors during primitive execution
Whenever something unrecoverable handles, this is called. Currently just returns the
error code from interpreter()
.
#define handle_error(code) do {return code;} while(0)
4.5.3 Stack Operation Assertion
These macros are used when popping and pushing elements on stacks, and check if the stack is empty or full.
#define assert_pop(sp,min,name,fail_reason) if (sp <= min) { print_error(name "stack underflow");BACKTRACE();handle_error(fail_reason);} #define assert_push(sp,min,size,fail_reason) if (sp > min+size){ print_error("stack overflow");BACKTRACE();handle_error(fail_reason);}
4.5.4 Stack operations
These macros are the basic internal stack manupulation functions. All stacks are assumed empty ascending. The following routines are general:
Peeking into the stack (as opposed to removing a value), currently only used for the
_dup
primitive
#define peek_n(sp,nth) (*(sp-nth))
Basic push and pop operations.
/* push value onto stack indicated by stack pointer sp */ #define push_(sp,val) *sp=val;sp++; /* pop value from stack indicated by stack pointer sp */ #define pop_(sp) --sp;*sp;
The rest is meant for the three specific stacks (TODO: link to stack explanation). The push functions push (safely) a value onto the respective stack. The pop functions pop (safely) a value from the respective stack.
<IP_ADDRESS> Data Stack (Parameter Stack)
#define ppush(val) ({assert_push(psp,pstack,VM_PSTACK,INTERNAL_ERROR_PSTACK_OFLOW);push_(psp,val)}) #define ppop() ({assert_pop(psp,pstack,"p",INTERNAL_ERROR_PSTACK_UFLOW);pop_(psp)})
<IP_ADDRESS> Return Stack
#define returnpush(val) ({assert_push(returnsp,returnstack,VM_RETURNSTACK,INTERNAL_ERROR_RSTACK_OFLOW);push_(returnsp,val)}) #define returnpop() ({assert_pop(returnsp,returnstack,"return",INTERNAL_ERROR_RSTACK_UFLOW);pop_(returnsp)})
<IP_ADDRESS> Retain Stack
#define retainpush(val) ({assert_push(retainsp,retainstack,VM_RETAINSTACK,INTERNAL_ERROR_RTSTACK_OFLOW);push_(retainsp,val)}) #define retainpop() ({assert_pop(retainsp,retainstack,"retain",INTERNAL_ERROR_RTSTACK_OFLOW);pop_(retainsp)})
4.6 Main Interpreter function interpreter()
Basic Flowchart
The interpreter()
functions is structured as follows:
int interpreter(short_jump_target start_address) { <<state_definition>> <<state_initialization>> while(1) { inst i; i = (*pc++); dispatch: <<debug_statements>> switch(i) { <<big_fat_switch>> default: printf("unimplemented instruction %#x\n",i); handle_error(INTERNAL_ERROR_INVALID_OPCODE); } goto end_inst; /* normal instructions skip the call execution paths */ nested_call: /* common execution path for non-tail calls */ <<nested_call_common>> tail_call: /* common execution path for tail calls */ <<tail_call_common>> end_inst: /* end of instruction processing */ (void) 0; } }
The labels after the switch statement are common excution paths of several primitives.
4.6.1 Internal State Definition
<IP_ADDRESS> Stacks
- data stack (or parameter stack) and pointer
- return stack and pointer
- retain stack and pointer
/* parameter stack */ static cell pstack[VM_PSTACK]={0}; static cell* psp; psp = &pstack[0]; /* return stack */ static return_entry returnstack[VM_RETURNSTACK]={{0}}; static return_entry* returnsp; returnsp = &returnstack[0]; /* retain stack */ static cell retainstack[VM_RETAINSTACK]={0}; static cell* retainsp; retainsp = &retainstack[0];
The data and retain stacks are using scalar elements of type cell
, while the return
stack holds structs.
The size macros are defined in the header file (see 4.1.2), and control the memory usage of the VM.
<IP_ADDRESS> Program counter
This is the most basic state for any program interpreting machine. It holds the address of the next primitive instruction in the instruction stream (bytecode image).
inst *pc; pc = &image[(start_address ? : START_WORD_OFFSET)]; /* point to the start of the program */
The initial value is either taken from the interpreter()
call as parameter, or by using
the START_WORD_OFFSET
macro from bytecode.h
, that translates the START_WORD
global
variable (see 1.3) into an address.
<IP_ADDRESS> Tail Call Optimization
The MFactor implementation relies on (explicit) tail call elimination. Because of the
nature of the language, the compiler can detect all cases of tail calls easily and
replaces the call
instructions with their tcall
equivalents:
bcall
->btcall
scall
->stcall
acall
->atcall
The tcall
instructions all amount to a jump to the called word, thus replacing the
current stack frame instead of adding one to the dynamic chain. Since Iteration is
implemented with (tail)-recursion, disabling tail calls severly limit loop execution, and
should only be turned on locally for debugging purposes.
This variable controls wether tail calls should be handled as such, or redirected to
nested calls:
static bool tailcall; tailcall = true; /* enable tail call jumping by default */
<IP_ADDRESS> Helper "Register"
To help manage jumping through the code, a variable x
is defined here and used in
various primitive instructions. Normally, the needed variables are defined locally in the
intruction cases.
cell x; /* temporary value for operations */
<IP_ADDRESS> Debug Mode
To debug internals, this debug_mode
state can be set (also via an isntruction, TODO:
link). If so, it outputs internal state before processing an instruction, and waits for a
keypress before processing it. The debug_nest
counter is used to keep track of return
stack levels to make sure debug mode is exited corretly and to provide nesting
information.
/* single step debugging*/ unsigned int debug_nest = 0; /* used in debug mode to track when * to stop single stepping*/ bool debug_mode = false; #if DEBUG debug_mode=true; #endif
4.6.2 State initialization
The initial of the vm: Program counter points to the starting word, data and retain stack empty, return stack contains a default frame.
<IP_ADDRESS> Output File Descriptor
The value passed to all libc stdio commands used during primitive instruction execution.
Ostream = stdout; /* print everything to stdout per default */
<IP_ADDRESS> Special Variable Initalization
See 4.3.2.
init_specials();
4.6.3 Primitive Instructions
All the primitives, that defined words consist of, are described here. Some of them are prepended with underscores to avoid name clashes with existing c functionality.
<IP_ADDRESS> Basic stack manipulation
drop
- remove value on top of data stack
dup
- (internally named _dup to avoid name conflits) duplicate the item on top of data stack
swap
- swap two top values on data stack
to_r
- move top of data stack to retain stack
r_from
- move top of retain stack to data stack
clear
- clear contents of data stack
case drop: ppop(); break; case _dup: ppush(peek_n(psp,1)); break; case swap: { x=ppop(); cell y = ppop(); ppush(x); ppush(y); } break; case to_r: retainpush(ppop()); break; case r_from: ppush(retainpop()); break; case clear: psp = &pstack[0]; break;
<IP_ADDRESS> Arithmetic Instructions
All arithmetic instructions either manipulate the value on top of the stack, or replace the two top values with the results. For defining unary and binary operations with C operators, two helper macros are used:
#define UNOP(op) { \ x=(op ((intptr_t) ppop())); \ ppush(x); \ } break #define BINOP(op) { \ x = ppop(); \ cell y = ppop(); \ ppush(((intptr_t)y) op ((intptr_t)x)); \ } break
The instructions themselves:
case add: BINOP(+); break; case mul: BINOP(*); break; case sub: BINOP(-); break; case neg: UNOP(-); break; case asl: BINOP(<<); break; /* arithmetic shift left */ case asr: BINOP(>>); break; /* arithmetic shift right */ case div: BINOP(/); break; case mod: BINOP(%); break; case bitand: BINOP(&); break; case bitor: BINOP(|); break; case bitxor: BINOP(^); break; case bitnot: UNOP(~); break; case gt: BINOP(>); break; case lt: BINOP(<); break; case eql: BINOP(==); break;
<IP_ADDRESS> Constants
These push constants onto the stack. These save space compared to pushing a literal onto the stack, which takes two bytes instead of one.
case zero: ppush(0); break; case one: ppush(1); break; case two: ppush(2); break;
<IP_ADDRESS> Access to Internal Constants
These provide information about some internals to the application:
- work memory bounds
- dictionary bounds
- size in bytes of stack elements (platform-dependent)
- starting instruction code
Note that the memrange and memstart/memend instructions are redundant (TODO: remove memrange)
case memstart: ppush((cell)memory); break; case memend: ppush((cell)(memory+VM_MEM)); break; case memrange: ppush((cell) memory); ppush((cell) memory+VM_MEM*sizeof(cell)); break; case dictstart: ppush((cell)dict); break; case dictend: ppush((cell)(dict+VM_DICT)); break; case cellsize: ppush((cell)sizeof(cell)); break; case instbase: ppush((cell)INSTBASE); break;
<IP_ADDRESS> Access to Internal State
These primitives expose internal state to the application.
Access to the stack pointers is needed for saving and storing state, and is mainly provided to be able to implement exception handling, which needs to store the values of these pointers(on the retain stack) for later recovery.
The number of the desired stack pointer is taken from top of stack.
case get_sp: switch (ppop()) { case 0 : ppush((cell)psp); break; case 1 : ppush((cell)retainsp); break; case 2 : ppush((cell)returnsp); break; default: ppush((cell)0); break; } break; case set_sp: { unsigned char n = (unsigned char)ppop(); cell* newsp = (cell*)ppop(); switch (n) { case 0 : psp = newsp; break; case 1 : retainsp = newsp; break; case 2 : returnsp = (return_entry *)newsp; break; } } break;
Access to the special variables is provided to control or query several aspects of interpreter state, see 4.3 for details. Number of special variable is taken from top of stack.
case get_special: { unsigned char i = (unsigned char)ppop(); if (i < NumSpecials) ppush(special_vars[i]); else { printf("illegal specials index: %d\n", i); handle_error(INTERNAL_ERROR_UNKNOWN_SPECIAL); } } break; case set_special: { unsigned char i = (unsigned char)ppop(); if (i < NumSpecials) special_vars[i] = ppop(); else { printf("illegal specials index: %d\n", i); handle_error(INTERNAL_ERROR_UNKNOWN_SPECIAL); } } break;
Push the current stack level on tos
case stack_level: ppush(psp-pstack); break;
<IP_ADDRESS> Literal Instructions
Literal instructions all push a value that is defined immediately after the instruction in the instruction stream, and push it onto the data stack. If the current program counter points to this:
| litb | 0x04 | inst1 | inst2 | … |
it will point to this after execution:
| inst1 | inst2 | … |
Because the literal byte is a "argument" to the litb instruction. The different literal instructions ahve different width arguments.
litb
pushes a byte-sized numberliti
pushes a cell-sized number
case litb: /* byte literal */ x=(cell)(*(pc++)); ppush(x); break; case ref: case liti: /* cell-wide literal */ x=*((cell *)pc); ppush(x); pc+=sizeof(cell); break;
litc
is meant for complex (non-scalar) literals like sequences. It expects its
argument to be in the format described in src/seq_headers.h. The only really
important thing about that format here is that the second byte specifies the length.
After execution of litc
, the address of that length-defining byte is left on the stack,
and the pc is adjustet to correctly skip all the inline data.
case litc: { /* complex inline literal */ /* pc is already at the next item -> header byte */ seq_header h = (seq_header)(*pc); ppush((cell)pc+1); /* leave address of count byte on stack */ pc += 2 + fe_seq_size(h,pc+1); } break;
oplit
is like litb
, and is intended for pushing an instruction on the stack literally,
instead of executing it. It is needed by the interactive bytecode-compiler to put
instructions onto the stack. It does however an additional transformation: since
primitives on the stack are no different from addresses in regard to being call
'ed,
the way to distinguish between a primitive to execute and an address to jump to is, that
primitives are represented as addresses that would be invalid otherwise. That is also the
reason for different values of INSTBASE
(see 4.1.2).
In essence:
| oplit | 0xa4 |
pushes 0xa4000000
onto the stack. Since this would be an invalid memory address on the
chosen architecture (checked during scall
and stcall
in <IP_ADDRESS>), it
is later known to be a primitive, not the location of a quotation when called.
case oplit: /* literal primitive operation */ x=(cell)*(pc++); ppush(x<<(8*(sizeof(cell)-sizeof(inst)))); break;
qstart
is pushing a literal quotation on the stack. Since all quotations are static
code, they are uniquely idenfified by that address. This address can then later be used
by a call instruction, namely scall
or stcall
. The implementation is not much
different from litc
, but quotations don't have a sequence header, only a count byte as
first element.
case qstart: { uint8_t l = *((uint8_t *)pc) + 1; ppush((cell)(pc + 1)); /* skip over quotation length and count byte, leaving pc after qend */ pc = pc + l + 1; } break;
<IP_ADDRESS> IO Primitives
Several primitives make calls to libc's stdio functionality. All output is directed to
Ostream
(see [[output-handling]).
emit
- output top of stack as
char
receive
- read(blocking) one character from stdin
pwrite
- "pretty-write", outputs top of stack with printf as decimal
pwritex
- "pretty-write-hex", outputs top of stack with printf as hex and
0x
prefix writex
- "write-hex", outputs top of stack with printf as hex
The output functions all check current_fd()
(see 4.4.1). If it is NULL, the
element not to be output at all and simply dropped.
case emit: if (current_fd() != NULL) { fputc(ppop(),current_fd()); fflush(stdout); /* TODO remove when flushing is delegated to * higher level calls */ } else (void) ppop(); break; case receive: ppush(read_char()); break; case _pwrite: if (current_fd() != NULL) fprintf(current_fd(), "%ld", ppop()); else (void) ppop(); break; case pwritex: if (current_fd() != NULL) fprintf(current_fd(), "%#lx", ppop()); else (void) ppop(); break; case writex: if (current_fd() != NULL) fprintf(current_fd(), "%lx", ppop()); else (void) ppop(); break;
<IP_ADDRESS> Call/Return Instructions
These instructions implement control flow, and make use of the return stack. The difference between the call instructions is how they get their target address, how wide that address is and wether or not they execute a jump (<IP_ADDRESS>).
| primitive | where to get address | type of address | tail-call |
|---|---|---|---|
| bcall | instruction stream | short (16 bit) relative | no |
| btcall | instruction stream | short (16 bit) relative | yes |
| acall | instruction stream | full size absolute | no |
| scall | top of stack | full size absolute | no |
| stcall | top of stack | full size absolute | yes |
case bcall: { /* base-relative call */ _bcall: x = (cell)(BASE + *((short_jump_target *)pc)); /* set the target */ pc += sizeof(short_jump_target); goto nested_call; } break; case btcall: /* base-relative tail-call, effectively a goto */ if (!tailcall) goto _bcall; x = (cell)(BASE + *((short_jump_target *)pc)); /* set the target */ goto tail_call; break; case acall: { /* absolute call */ x = (cell) *((jump_target *)pc); /* set the target */ pc += sizeof(jump_target); goto nested_call; } break; case scall: _scall: /* check if call target is primitive, if yes, substitute execution (tail call), since call only applies to quotations */ x = ppop(); /* set the target */ if (x >= INSTBASE_CELL) { i = (x >> (8 * (sizeof(inst *) - sizeof(inst)))); goto dispatch; /* calling a primitive "substitutes" the call with the primitive */ } else { goto nested_call; } break; case stcall: if (!tailcall) goto _scall; x = ppop(); /* set the target */ if (x >= INSTBASE_CELL) { i=( x >> (8 * (sizeof(inst*) - sizeof(inst)))); goto dispatch; /* already a tail call */ } else { goto tail_call; } break;
All the above jump outside of the switch statement, either to the tailcall
or
nestedcall
label. These common execution paths are described next.
For normal nested calls, a stack frame is created that contains the return address and a
reference to the word being called. This second item is not needed for execution, but
helps debugging since it always shows where the currently executed word was started. If
debug mode is on, increase the nesting counter debug_nest
.
Execution path nested_call
:
{ inst *next_word = (inst *) x; /* was set by the call instructions that jumped here */ return_entry e = {.return_address = pc, .current_call=next_word}; returnpush(e); if (debug_mode) { char * name = find_by_address(next_word); debug_nest++; printf("calling: %s -> %d\n",name,debug_nest); } pc=next_word; } goto end_inst;
For tail calls, the current word execution is just substituted (equivalent to a simple jmp
)
Execution path tail_call
:
{ inst *next_word = (inst *) x; /* was set by the call instructions that jumped here */ if (debug_mode) { char * name = find_by_address(next_word); printf("tail calling: %s -> %d\n",name,debug_nest); } pc = next_word; } goto end_inst;
qend
, "quotation end", is the return instruction. It pops a frame of the return stack
and sets the program counter to resume execution after the original call site. Again, in
case of debug mode being activated, record the nesting level change.
case qend: { return_entry e = returnpop(); if (debug_mode) { if (debug_nest > 0) { printf("<- %d\n",debug_nest); debug_nest--; } else { debug_mode=false; } } pc=e.return_address; } break;
<IP_ADDRESS> Conditional
There is only one instruction for handling conditionals, which implements factor word ?
directly.
: ? ( ? true false -- true/false )
If the flag (on third position) is false, leave only the true
value, otherwise leave the
false
value.
case truefalse: { cell false_cons = ppop(); cell true_cons = ppop(); cell cond = ppop(); ppush(cond ? true_cons : false_cons); } break;
<IP_ADDRESS> Memory Access
These primitives write/read single values to/from memory. Two are for byte-sized writes, and the other two for cell-sized writes.
case setmem: x = ppop(); *((cell*)x) = (ppop()); break; case setmem8: x=ppop(); *((char*)x) = ((ppop() & 0xff)); break; case getmem: { cell *addr = (cell *)ppop(); x = *addr; ppush(x); } break; case getmem8: { char *addr = (char *)ppop(); x = (cell)(*(addr)); ppush(x); } break;
<IP_ADDRESS> Foreign Function Execution
For using foreign functions(see 1.5), these instructions are provided.
ff
takes an index from the stack that points in to the FF_Table
, which is generated in
bytecode.c
along the image and the dictionary from the ff.yml
specification. This
table holds all the function addresses that have been made available. ff
returns this
address.
The mfactor code that actually makes use of these primitives is generated in
generated/ff.mfactor
.
case ff: { #ifndef FF_LENGTH #define FF_LENGTH 0 #endif unsigned int i = ppop(); if (i >= FF_LENGTH) { printf("no ff entry with index %i\n",i); BACKTRACE(); handle_error(INTERNAL_ERROR_UNKNOWN_FF); } ppush((cell)FF_Table[i]); } break;
The ccall*
primitives are then used to actually call the function pointed to by
above-obtained address. Instead of providing support for each architcture's full calling
convention, the c compiler is used to generate correct call frames by casting the function
address to one more common function prototypes. The name of the ccall_*
indicates which
calling convention should be used.
All these primitives return a value, even if the underlying c function is declared with a
void
return type. It is up to the generated code in generated/ff.mfactor
to correctly
drop these junk values.
This is a speed/space tradeoff, since all these instructions do similar things, but computing a call frame for the underlying call convention would be more expensive, in addition to somehow providing the information about the argument number and types.
case ccall_i: { int(*fun)(int) = (int (*)(int))ppop(); int i1 = (int)ppop(); int res = fun(i1); ppush((cell)res); } break; case ccall_s: { int(*fun)(short) = (int (*)(short))ppop(); short s1 = (short)ppop(); int res = fun(s1); ppush((cell)res); } break; case ccall_b: { int(*fun)(char) = (int (*)(char))ppop(); int b1 = (char)ppop(); int res = fun(b1); ppush((cell)res); } break; case ccall_bi: { int(*fun)(char, int) = (int (*)(char,int))ppop(); int i2 = (int)ppop(); char b1 = (char)ppop(); int res = fun(b1,i2); ppush((cell)res); } break; case ccall_ib: { int(*fun)(int, char) = (int (*)(int, char))ppop(); char b2 = (char)ppop(); int i1 = (int)ppop(); int res = fun(i1,b2); ppush((cell)res); } break; case ccall_is: { int(*fun)(int,short) = (int (*)(int,short))ppop(); short s2 = (short)ppop(); int i1 = (int)ppop(); int res = fun(i1,s2); ppush((cell)res); } break; case ccall_iis: { int(*fun)(int,int,short) = (int (*)(int,int,short))ppop(); short s3 = (short)ppop(); int i2 = (int)ppop(); int i1 = (int)ppop(); int res = fun(i1,i2,s3); ppush((cell)res); } break; case ccall_iii: { int(*fun)(int,int,int) = (int (*)(int,int,int))ppop(); int i3 = (int)ppop(); int i2 = (int)ppop(); int i1 = (int)ppop(); int res = fun(i1,i2,i3); ppush((cell)res); } break; case ccall_ii: { int(*fun)(int,int) = (int (*)(int,int))ppop(); int i2 = (int)ppop(); int i1 = (int)ppop(); int res = fun(i1,i2); ppush((cell)res); } break; case ccall_v: { int(*fun)(void) = (int (*)(void))ppop(); int res = fun(); ppush((cell)res); } break; case ccall_lit: break; /* address is literal, nothing happens */
<IP_ADDRESS> (As of Yet) Undocumented instructions
case nop: break; case dref: x=(cell)(*((short_jump_target*) pc)); ppush((cell)memory+x); pc += sizeof(short_jump_target); break; case bref: /* reference to short-length in-memory data (type can be seen on-site)*/ case blitq: /* deprecated */ x=(cell)(*((short_jump_target*) pc)); ppush((cell) (BASE + ((short_jump_target) x))); pc += sizeof(short_jump_target); break; case quit: /* quit the interpreter, returning 0 */ printf("quitting interpreter!\n"); return 0; case token: { /* get one token from standard input */ char *tok = read_token(); if (tok) { ppush((cell)tok); } else { print_error("token reader error"); handle_error(INTERNAL_ERROR_TOKEN_READ); }} break; case lookup_name: { /* provide a search start for the name (addr length) by using * the hash table lookup */ cell length = ppop(); cell name = ppop(); ppush(lookup_ht_entry((uint8_t) length,(char *) name)); } break; case error: /* trigger internal error (bypasses any exception handling) */ printf("error!\n"); printf("\np"); printstack(psp,pstack); printf("retain"); printstack(retainsp,retainstack); printf("return"); print_return_stack(returnsp,returnstack,(inst *)BASE); BACKTRACE(); handle_error(INTERNAL_ERROR_GENERAL); break; case tstart: /* start the timer */ start_timer(); break; case tend: /* end the timer, return measurement results ( -- usecs secs ) */ { unsigned int sec,usec; end_timer(&sec,&usec); ppush(usec); ppush(sec); } break; case parsenum: { /* parse the (c)string at the address on top of stack as number */ char *str = (char *)ppop(); cell num = 0xa5a5a5a5; bool success=parse_number(str+1,&num); ppush(success ? num : (cell) str); ppush((cell)success); } break; case tail: /* activate tail call elimination */ tailcall=true; break; case notail: /* deactivate tail call elimination */ tailcall=false; break; case reset: /* call externally refined system-reset */ reset_system(); break; case debug: /* activate debug mode */ if (!debug_mode) { debug_mode=true; } break;
5 Appendix: Complete Sources
5.1 interpreter.h
#ifndef INTERPRETER_H #define INTERPRETER_H #include <stdbool.h> #include <stdint.h> #include "generated/bytecode.h" typedef unsigned char inst; typedef unsigned short short_jump_target; /* relative jumps in 64k on 32 bit */ typedef uintptr_t jump_target; /* long absolute jump */ typedef uintptr_t cell; /* memory cell must at least hold pointer */ typedef struct dict_entry { inst * address; /* pointer into memory */ unsigned char flags; /* may include other flags later (inline, recursive, etc) */ unsigned char name_header; /* should always be zero */ unsigned char name_length; char name[]; } __attribute__((packed)) dict_entry; /* data memory (affects non-transient data) in cells*/ #ifndef VM_MEM #define VM_MEM 256 #endif /* dictionary size (affects number of named items)*/ #ifndef VM_DICT #define VM_DICT 512 #endif /* parameter stack size (affects transient data)*/ #ifndef VM_PSTACK #define VM_PSTACK 64 #endif /* return stack size (affects nesting of functions)*/ #ifndef VM_RETURNSTACK #define VM_RETURNSTACK 64 #endif /* retain stack size (affects maximum amount of postponing data use) */ #ifndef VM_RETAINSTACK #define VM_RETAINSTACK 32 #endif #ifndef INSTBASE #if (__linux && __LP64__) #define INSTBASE 0x80U #elif (CORTEX_M) #define INSTBASE 0xA0U #else #error "don't know instruction code base for architecure!" #endif #endif #define INSTBASE_CELL ((cell)INSTBASE<<(8*(sizeof(inst *)-sizeof(inst)))) extern inst image[]; extern dict_entry dict[VM_DICT]; extern uint16_t dict_hash_index[]; extern cell FF_Table[]; /* these values can be returned from interpreter() */ #define INTERNAL_ERROR_PSTACK_UFLOW -1 #define INTERNAL_ERROR_PSTACK_OFLOW -2 #define INTERNAL_ERROR_RSTACK_UFLOW -3 #define INTERNAL_ERROR_RSTACK_OFLOW -4 #define INTERNAL_ERROR_RTSTACK_UFLOW -5 #define INTERNAL_ERROR_RTSTACK_OFLOW -6 #define INTERNAL_ERROR_INVALID_OPCODE -7 #define INTERNAL_ERROR_MEM_FAULT -8 #define INTERNAL_ERROR_UNKNOWN_FF -9 #define INTERNAL_ERROR_TOKEN_READ -10 #define INTERNAL_ERROR_GENERAL -11 #define INTERNAL_ERROR_UNKNOWN_SPECIAL -12 int interpreter(short_jump_target); #endif
5.2 interpreter.c
#include "interpreter.h" #include "reader.h" #include <string.h> #include <stddef.h> #include <stdio.h> #include <stdbool.h> /* these are available after mfactor task has run */ #include "generated/bytecode.h" #include "generated/inst_enum.h" /* target specific stuff */ #include "runtime.h" #include "reset_system.h" #include "seq_headers.h" /* main memory to work with */ static cell memory[VM_MEM]; #define _NumSpecials 10 static const unsigned char NumSpecials = _NumSpecials; static cell special_vars[_NumSpecials]; /* currently implemented special variables 0: MP - memory write pointer 1: HANDLER - handler frame location in r(etain) stack (dynamic chain for catch frames) 2: DEBUG_LEVEL - 0 to turn off, increasing will produce more verbose debug output 3: RESTART - word where to restart when hard error occured 4: STEP_HANDLER - address of handler which can be used for single stepping 5: BASE - address of current 64k segment base 6: OUTPUT_STREAM: 1: stdout, 2: stderr, 3: null */ #define MP special_vars[0] #define HANDLER special_vars[1] #define DEBUG_LEVEL special_vars[2] #define BASE special_vars[5] #define OUTPUT_STREAM special_vars[6] /* known stream descriptors for OUTPUT_STREAM */ #define STDOUT 1 #define STDERR 2 #define NULLOUT 3 static void init_specials() { HANDLER = 0; MP = (cell)memory; /* start of user memory */ BASE = (cell)ℑ /* start of bytecode segment */ OUTPUT_STREAM = STDOUT; /* output to standard output per default */ } static FILE * Ostream; /* used by reporting functions, so they can temporarily print to different file descriptor */ /* get the current stdio FILE from the special variable, or NULL if unknown or muted by * choosing NULLOUT */ static FILE * current_fd(void) { if (OUTPUT_STREAM == 2) return stderr; else if (OUTPUT_STREAM == 1) return stdout; else return NULL; } typedef struct return_entry { inst * return_address; inst * current_call; } return_entry; uint32_t lookup_ht_entry(uint8_t length, char* name) { uint32_t hash = 5381; for (int i = 0; i < length; i++) { hash = hash * 33 + name[i]; } return (cell)dict+dict_hash_index[hash%256]; } /* get the name of the word, only for debugging */ /* probably fails for non-null-terminated strings */ static char* find_by_address( inst * word) { static char notfound[] = "(internal or private)"; for (char * ptr=(char*)dict; (ptr < ((char*)dict+sizeof(dict)))&&(((dict_entry*)ptr)->name_length > 0); ptr += (((dict_entry*)ptr)->name_length + 3*sizeof(unsigned char) + sizeof(void*))) { dict_entry *dptr = (dict_entry*)ptr; if (dptr->address == word) return dptr->name; } return notfound; } static bool parse_number(char *str, cell * number){ int num; unsigned int read = sscanf(str,"%i",&num); if (read == 1) { *number = (cell)num; return true; } else { return false; } } /* display data stack entries */ static void printstack(cell * sp, cell * stack) { fprintf(Ostream, "stack:"); for(cell* ptr = stack;ptr < sp;ptr++) { fprintf(Ostream, " %#lx",*ptr); } fprintf(Ostream, "\n"); } /* display return stack entries */ static void print_return_stack(return_entry * sp, return_entry * stack, inst * base) { fprintf(Ostream, "stack:"); for(return_entry* ptr = sp-1;ptr >= stack;ptr--) { fprintf(Ostream, " {%#lx->%#lx}",((uintptr_t)ptr->current_call-(uintptr_t)base), ((uintptr_t)ptr->return_address)-(uintptr_t) base); } fprintf(Ostream, "\n"); } /* print a backtrace of the return stack */ static void backtrace(return_entry * sp, return_entry * stack, inst * base, inst * pc) { fprintf(Ostream, "backtrace @ %#lx:\n",(uintptr_t)(pc-base)); for(return_entry* ptr = sp-1;ptr >= stack;ptr--) { char *current_name = find_by_address(ptr->current_call); fprintf(Ostream, "%#lx %s\n",(uintptr_t)(ptr->current_call - base),current_name); } } static void print_error(char * str) { fprintf(stderr, "error: "); fprintf(stderr, str); fprintf(stderr, "\n"); } #define BACKTRACE() do { \ FILE * old_out = Ostream; \ Ostream = stderr; \ printstack(psp,pstack); \ printstack(retainsp,retainstack); \ backtrace(returnsp,returnstack,(inst *)BASE,pc); \ Ostream = old_out; \ } while (0) #define handle_error(code) do {return code;} while(0) #define assert_pop(sp,min,name,fail_reason) if (sp <= min) { print_error(name "stack underflow");BACKTRACE();handle_error(fail_reason);} #define assert_push(sp,min,size,fail_reason) if (sp > min+size){ print_error("stack overflow");BACKTRACE();handle_error(fail_reason);} #define peek_n(sp,nth) (*(sp-nth)) /* push value onto stack indicated by stack pointer sp */ #define push_(sp,val) *sp=val;sp++; /* pop value from stack indicated by stack pointer sp */ #define pop_(sp) --sp;*sp; #define ppush(val) ({assert_push(psp,pstack,VM_PSTACK,INTERNAL_ERROR_PSTACK_OFLOW);push_(psp,val)}) #define ppop() ({assert_pop(psp,pstack,"p",INTERNAL_ERROR_PSTACK_UFLOW);pop_(psp)}) #define returnpush(val) ({assert_push(returnsp,returnstack,VM_RETURNSTACK,INTERNAL_ERROR_RSTACK_OFLOW);push_(returnsp,val)}) #define returnpop() ({assert_pop(returnsp,returnstack,"return",INTERNAL_ERROR_RSTACK_UFLOW);pop_(returnsp)}) #define retainpush(val) ({assert_push(retainsp,retainstack,VM_RETAINSTACK,INTERNAL_ERROR_RTSTACK_OFLOW);push_(retainsp,val)}) #define retainpop() ({assert_pop(retainsp,retainstack,"retain",INTERNAL_ERROR_RTSTACK_OFLOW);pop_(retainsp)}) int interpreter(short_jump_target start_address) { /* parameter stack */ static cell pstack[VM_PSTACK]={0}; static cell* psp; psp = &pstack[0]; /* return stack */ static return_entry returnstack[VM_RETURNSTACK]={{0}}; static return_entry* returnsp; returnsp = &returnstack[0]; /* retain stack */ static cell retainstack[VM_RETAINSTACK]={0}; static cell* retainsp; retainsp = &retainstack[0]; inst *pc; pc = &image[(start_address ? : START_WORD_OFFSET)]; /* point to the start of the program */ static bool tailcall; tailcall = true; /* enable tail call jumping by default */ cell x; /* temporary value for operations */ /* single step debugging*/ unsigned int debug_nest = 0; /* used in debug mode to track when * to stop single stepping*/ bool debug_mode = false; #if DEBUG debug_mode=true; #endif Ostream = stdout; /* print everything to stdout per default */ init_specials(); while(1) { inst i; i = (*pc++); dispatch: switch(i) { case drop: ppop(); break; case _dup: ppush(peek_n(psp,1)); break; case swap: { x=ppop(); cell y = ppop(); ppush(x); ppush(y); } break; case to_r: retainpush(ppop()); break; case r_from: ppush(retainpop()); break; case clear: psp = &pstack[0]; break; #define UNOP(op) { \ x=(op ((intptr_t) ppop())); \ ppush(x); \ } break #define BINOP(op) { \ x = ppop(); \ cell y = ppop(); \ ppush(((intptr_t)y) op ((intptr_t)x)); \ } break case add: BINOP(+); break; case mul: BINOP(*); break; case sub: BINOP(-); break; case neg: UNOP(-); break; case asl: BINOP(<<); break; /* arithmetic shift left */ case asr: BINOP(>>); break; /* arithmetic shift right */ case div: BINOP(/); break; case mod: BINOP(%); break; case bitand: BINOP(&); break; case bitor: BINOP(|); break; case bitxor: BINOP(^); break; case bitnot: UNOP(~); break; case gt: BINOP(>); break; case lt: BINOP(<); break; case eql: BINOP(==); break; case zero: ppush(0); break; case one: ppush(1); break; case two: ppush(2); break; case memstart: ppush((cell)memory); break; case memend: ppush((cell)(memory+VM_MEM)); break; case memrange: ppush((cell) memory); ppush((cell) memory+VM_MEM*sizeof(cell)); break; case dictstart: ppush((cell)dict); break; case dictend: ppush((cell)(dict+VM_DICT)); break; case cellsize: ppush((cell)sizeof(cell)); break; case instbase: ppush((cell)INSTBASE); break; case get_sp: switch (ppop()) { case 0 : ppush((cell)psp); break; case 1 : ppush((cell)retainsp); break; case 2 : ppush((cell)returnsp); break; default: ppush((cell)0); break; } break; case set_sp: { unsigned char n = (unsigned char)ppop(); cell* newsp = (cell*)ppop(); switch (n) { case 0 : psp = newsp; break; case 1 : retainsp = newsp; break; case 2 : returnsp = (return_entry *)newsp; break; } } break; case get_special: { unsigned char i = (unsigned char)ppop(); if (i < NumSpecials) ppush(special_vars[i]); else { printf("illegal specials index: %d\n", i); handle_error(INTERNAL_ERROR_UNKNOWN_SPECIAL); } } break; case set_special: { unsigned char i = (unsigned char)ppop(); if (i < NumSpecials) special_vars[i] = ppop(); else { printf("illegal specials index: %d\n", i); handle_error(INTERNAL_ERROR_UNKNOWN_SPECIAL); } } break; case stack_level: ppush(psp-pstack); break; case litb: /* byte literal */ x=(cell)(*(pc++)); ppush(x); break; case ref: case liti: /* cell-wide literal */ x=*((cell *)pc); ppush(x); pc+=sizeof(cell); break; case litc: { /* complex inline literal */ /* pc is already at the next item -> header byte */ seq_header h = (seq_header)(*pc); ppush((cell)pc+1); /* leave address of count byte on stack */ pc += 2 + fe_seq_size(h,pc+1); } break; case oplit: /* literal primitive operation */ x=(cell)*(pc++); ppush(x<<(8*(sizeof(cell)-sizeof(inst)))); break; case qstart: { uint8_t l = *((uint8_t *)pc) + 1; ppush((cell)(pc + 1)); /* skip over quotation length and count byte, leaving pc after qend */ pc = pc + l + 1; } break; case emit: if (current_fd() != NULL) { fputc(ppop(),current_fd()); fflush(stdout); /* TODO remove when flushing is delegated to * higher level calls */ } else (void) ppop(); break; case receive: ppush(read_char()); break; case _pwrite: if (current_fd() != NULL) fprintf(current_fd(), "%ld", ppop()); else (void) ppop(); break; case pwritex: if (current_fd() != NULL) fprintf(current_fd(), "%#lx", ppop()); else (void) ppop(); break; case writex: if (current_fd() != NULL) fprintf(current_fd(), "%lx", ppop()); else (void) ppop(); break; case bcall: { /* base-relative call */ _bcall: x = (cell)(BASE + *((short_jump_target *)pc)); /* set the target */ pc += sizeof(short_jump_target); goto nested_call; } break; case btcall: /* base-relative tail-call, effectively a goto */ if (!tailcall) goto _bcall; x = (cell)(BASE + *((short_jump_target *)pc)); /* set the target */ goto tail_call; break; case acall: { /* absolute call */ x = (cell) *((jump_target *)pc); /* set the target */ pc += sizeof(jump_target); goto nested_call; } break; case scall: _scall: /* check if call target is primitive, if yes, substitute execution (tail call), since call only applies to quotations */ x = ppop(); /* set the target */ if (x >= INSTBASE_CELL) { i = (x >> (8 * (sizeof(inst *) - sizeof(inst)))); goto dispatch; /* calling a primitive "substitutes" the call with the primitive */ } else { goto nested_call; } break; case stcall: if (!tailcall) goto _scall; x = ppop(); /* set the target */ if (x >= INSTBASE_CELL) { i=( x >> (8 * (sizeof(inst*) - sizeof(inst)))); goto dispatch; /* already a tail call */ } else { goto tail_call; } break; case qend: { return_entry e = returnpop(); if (debug_mode) { if (debug_nest > 0) { printf("<- %d\n",debug_nest); debug_nest--; } else { debug_mode=false; } } pc=e.return_address; } break; case truefalse: { cell false_cons = ppop(); cell true_cons = ppop(); cell cond = ppop(); ppush(cond ? true_cons : false_cons); } break; case setmem: x = ppop(); *((cell*)x) = (ppop()); break; case setmem8: x=ppop(); *((char*)x) = ((ppop() & 0xff)); break; case getmem: { cell *addr = (cell *)ppop(); x = *addr; ppush(x); } break; case getmem8: { char *addr = (char *)ppop(); x = (cell)(*(addr)); ppush(x); } break; case ff: { #ifndef FF_LENGTH #define FF_LENGTH 0 #endif unsigned int i = ppop(); if (i >= FF_LENGTH) { printf("no ff entry with index %i\n",i); BACKTRACE(); handle_error(INTERNAL_ERROR_UNKNOWN_FF); } ppush((cell)FF_Table[i]); } break; case ccall_i: { int(*fun)(int) = (int (*)(int))ppop(); int i1 = (int)ppop(); int res = fun(i1); ppush((cell)res); } break; case ccall_s: { int(*fun)(short) = (int (*)(short))ppop(); short s1 = (short)ppop(); int res = fun(s1); ppush((cell)res); } break; case ccall_b: { int(*fun)(char) = (int (*)(char))ppop(); int b1 = (char)ppop(); int res = fun(b1); ppush((cell)res); } break; case ccall_bi: { int(*fun)(char, int) = (int (*)(char,int))ppop(); int i2 = (int)ppop(); char b1 = (char)ppop(); int res = fun(b1,i2); ppush((cell)res); } break; case ccall_ib: { int(*fun)(int, char) = (int (*)(int, char))ppop(); char b2 = (char)ppop(); int i1 = (int)ppop(); int res = fun(i1,b2); ppush((cell)res); } break; case ccall_is: { int(*fun)(int,short) = (int (*)(int,short))ppop(); short s2 = (short)ppop(); int i1 = (int)ppop(); int res = fun(i1,s2); ppush((cell)res); } break; case ccall_iis: { int(*fun)(int,int,short) = (int (*)(int,int,short))ppop(); short s3 = (short)ppop(); int i2 = (int)ppop(); int i1 = (int)ppop(); int res = fun(i1,i2,s3); ppush((cell)res); } break; case ccall_iii: { int(*fun)(int,int,int) = (int (*)(int,int,int))ppop(); int i3 = (int)ppop(); int i2 = (int)ppop(); int i1 = (int)ppop(); int res = fun(i1,i2,i3); ppush((cell)res); } break; case ccall_ii: { int(*fun)(int,int) = (int (*)(int,int))ppop(); int i2 = (int)ppop(); int i1 = (int)ppop(); int res = fun(i1,i2); ppush((cell)res); } break; case ccall_v: { int(*fun)(void) = (int (*)(void))ppop(); int res = fun(); ppush((cell)res); } break; case ccall_lit: break; /* address is literal, nothing happens */ case nop: break; case dref: x=(cell)(*((short_jump_target*) pc)); ppush((cell)memory+x); pc += sizeof(short_jump_target); break; case bref: /* reference to short-length in-memory data (type can be seen on-site)*/ case blitq: /* deprecated */ x=(cell)(*((short_jump_target*) pc)); ppush((cell) (BASE + ((short_jump_target) x))); pc += sizeof(short_jump_target); break; case quit: /* quit the interpreter, returning 0 */ printf("quitting interpreter!\n"); return 0; case token: { /* get one token from standard input */ char *tok = read_token(); if (tok) { ppush((cell)tok); } else { print_error("token reader error"); handle_error(INTERNAL_ERROR_TOKEN_READ); }} break; case lookup_name: { /* provide a search start for the name (addr length) by using * the hash table lookup */ cell length = ppop(); cell name = ppop(); ppush(lookup_ht_entry((uint8_t) length,(char *) name)); } break; case error: /* trigger internal error (bypasses any exception handling) */ printf("error!\n"); printf("\np"); printstack(psp,pstack); printf("retain"); printstack(retainsp,retainstack); printf("return"); print_return_stack(returnsp,returnstack,(inst *)BASE); BACKTRACE(); handle_error(INTERNAL_ERROR_GENERAL); break; case tstart: /* start the timer */ start_timer(); break; case tend: /* end the timer, return measurement results ( -- usecs secs ) */ { unsigned int sec,usec; end_timer(&sec,&usec); ppush(usec); ppush(sec); } break; case parsenum: { /* parse the (c)string at the address on top of stack as number */ char *str = (char *)ppop(); cell num = 0xa5a5a5a5; bool success=parse_number(str+1,&num); ppush(success ? num : (cell) str); ppush((cell)success); } break; case tail: /* activate tail call elimination */ tailcall=true; break; case notail: /* deactivate tail call elimination */ tailcall=false; break; case reset: /* call externally refined system-reset */ reset_system(); break; case debug: /* activate debug mode */ if (!debug_mode) { debug_mode=true; } break; default: printf("unimplemented instruction %#x\n",i); handle_error(INTERNAL_ERROR_INVALID_OPCODE); } goto end_inst; /* normal instructions skip the call execution paths */ nested_call: /* common execution path for non-tail calls */ { inst *next_word = (inst *) x; /* was set by the call instructions that jumped here */ return_entry e = {.return_address = pc, .current_call=next_word}; returnpush(e); if (debug_mode) { char * name = find_by_address(next_word); debug_nest++; printf("calling: %s -> %d\n",name,debug_nest); } pc=next_word; } goto end_inst; tail_call: /* common execution path for tail calls */ { inst *next_word = (inst *) x; /* was set by the call instructions that jumped here */ if (debug_mode) { char * name = find_by_address(next_word); printf("tail calling: %s -> %d\n",name,debug_nest); } pc = next_word; } goto end_inst; end_inst: /* end of instruction processing */ (void) 0; } }


VR Rpg Experience created in Unity3D, this small project has a medium size map that can be explored together with a Spell system and a Weapon Attack System. The spell system uses a neural network to analyse the runes drawn in the air and classify it so that later if the correct combination has been detected allow the user to cast a spell at enemies or random positions. The neural network has been trained with some data already allowing the users to test the system out without needing to wait. The accuracy of the network is not very high, with an accuracy of 70-80% however, this can be increased by getting more train data from different users. The attack systems with weapons use different information gathered over the last frames when hitting an enemy to calculate damage; Some of the data that will be taken into consideration is the max velocity, velocity when entering the body, velocity when exiting the body (if the user exits), swing angle. This data is then applied to the base damage of the weapon to get the end damage.


Register
Login
Hey Buddy, ready to take your adventure?
By click on this box you agree with Meotrics’s
Policy
and
Terms
I am ready
Already got Meotrics accournt?
Login
here
Hey Buddy, welcome back!
Login
Not yet got Meotrics account?
Register
here


Y.mojito.Client Class
The starting point for mojito to run in the browser. You can access one instance of the Mojito Client running within the browser environment through window.YMojito.client.
Item Index
Methods
attachBinders
-
binderMap
-
parentId
-
topLevelMojitViewId
Given a set of binder information, initialize binder instances and bind them to the page.
Parameters:
-
binderMap
ObjectviewId ==> binder data, contains all we need from the mojit dispatch's meta object about all the binders that were executed to create the DOM addition recently added to the document.
-
parentId
Stringthe parent binder view id to attach any children.
-
topLevelMojitViewId
Stringthe topmost (root) binder view id to attach as a child to the parent.
destroyMojitProxy
-
id
-
retainNode
Parameters:
-
id
StringThe mojit's viewId to destroy
-
retainNode
Boolean
executeAction
-
command
-
viewId
-
cb
Used for binders to execute their actions through the Mojito framework through their proxies.
Parameters:
-
command
Objectmust contain mojit id and action to execute.
-
viewId
Stringthe view id of the current mojit, which is executing the action.
-
cb
Functioncallback to run when complete.
fireLifecycle
-
evt
-
data
Fires a lifecycle event.
Parameters:
-
evt
StringThe name of event to fire.
-
data
ObjectThe data to pass to listeners.
pause
()
Pause the Mojito Client and all mojits that are running. This will notify all binders that they have been paused by calling their onPause() functions. It will prevent the immediate execution of several mojit proxy operations that might cause a long process to begin (especially things that might go to the server).
To resume, simply call .resume(). This will immediately execute all actions that occurred while Mojito was paused.
resume
()
Resumes the Mojito client after it has been paused (see method "pause"). If there are any queued actions that were executed and cached during the pause, calling resume() will immediately execute them. All binders are notified through their onResume() function that they are been resumed.
subscribe
-
evt
-
cb
Subscribe to a MojitoClient lifecycle event.
Parameters:
-
evt
Stringname of event to subscribe to.
-
cb
Function(data)callback called when the event fires.


Fits singular linear models to longitudinal data. Singular linear models are useful when the number, or timing, of longitudinal observations may be informative about the observations themselves. They are described in Farewell (2010) <doi:10.1093/biomet/asp068>, and are extensions of the linear increments model <doi:10.1111/j.1467-9876.2007.00590.x> to general longitudinal data.
| Version: | 0.1.1 |
| Depends: | R (≥ 3.2.0), data.table (≥ 1.9.6) |
| Imports: | stats, MASS (≥ 7.3) |
| Suggests: | lme4 (≥ 1.1), jmcm (≥ <IP_ADDRESS>), gee (≥ 4.13-19), ggplot2 (≥ 2.1.0), testthat (≥ 1.0.2), knitr, rmarkdown |
| Published: | 2017-05-15 |
| Author: | Daniel Farewell [aut, cre] |
| Maintainer: | Daniel Farewell <farewelld at cf.ac.uk> |
| License: | GPL-3 |
| NeedsCompilation: | no |
| Materials: | NEWS |
| CRAN checks: | slim results |
| Reference manual: | slim.pdf |
| Vignettes: |
slim: Singular Linear Models |
| Package source: | slim_0.1.1.tar.gz |
| Windows binaries: | r-devel: slim_0.1.1.zip, r-release: slim_0.1.1.zip, r-oldrel: slim_0.1.1.zip |
| OS X binaries: | r-release: slim_0.1.1.tgz, r-oldrel: slim_0.1.1.tgz |
| Old sources: | slim archive |
Please use the canonical form https://CRAN.R-project.org/package=slim to link to this page.


Containerization and virtualization are two ways to deploy multiple, isolated services on a single platform.
Containerization:
Containerization is a lightweight alternative to full machine virtualization that involves encapsulating an
application in a container with its own operating environment.Containers use the same host operating system (OS)
repeatedly, instead of installing an OS for each guest VM. Containers also provide a way to isolate applications and
provide a virtual platform for applications to run on . Two main differences exist between a container and a
hypervisor system.
Virtualization:
A virtual machine is a copy of a complete server basically it has its own OS which means the replication of
binaries, drivers, or libraries between different VMs running on the same server and a wastage of significant server
resources.
Containerization vs Virtualization:
Containerization provides an isolation to run your application while sharing the same OS resources and in the other
hand Virtualization provides the same flavour but it have its own OS so VM systems can run different operating
systems and VM will host multiple applications whose mix may change over time versus a container that will normally
have a single application.
In conclusion we can say if we have a 2 Node API running on a Different Node Version it may be possible if we deploy
them in the same VM they may Break but if we deploy them in a Container they are completely isolated with each
other.
So basically the use case can be if we want two Environments Test and TestMock we can create two virtual machines on
the same server and deploy our packages in a standalone container.


Labour refuses to rule out unlimited EU immigration in Brexit deal
by Stefan BosciaLabour’s election coordinator has refused to deny the party would continue to allow uncontrolled immigration from the EU in its prospective Brexit deal.
Labour’s Brexit policy, if it wins the election, is to negotiate a new deal with the EU and put it to the public in a confirmatory referendum.
However, the party’s election coordinator, and shadow communities secretary, Andrew Gwynne today refused to say whether the deal would end freedom of movement with the other 27 EU member states.
In the party’s 2017 manifesto, Labour said it would end unfettered immigration with the EU as a part of its Brexit deal.
Gwynne told the BBC: “Let me be very clear, we are looking for reciprocal agreements with the EU 27 that allows British citizens to enjoy some of the freedoms they will lose as a result of Brexit.
“Things like Erasmus [university exchange program], things like being able to continue working and living in EU states.
“The quid pro quo is those EU citizens in the UK….will be able to as well.”
The prevarication comes on the back of Labour leader Jeremy Corbyn also refusing to rule out an end to freedom of movement at a campaign event.
Corbyn is set to meet with senior Labour figures on Saturday to finalise the party’s election manifesto.
“I want to make sure all those EU nationals remain here, can come here, can stay here,” he said.
Labour’s Brexit policy has been a contentious issue, with many accusing of the party of sitting on the fence.
Members finally voted to make a second Brexit referendum party policy at its annual conference in September.
Members voted that if Labour wins government it will negotiate a new Brexit deal with the EU, put that deal to a second referendum against a remain option and not officially campaign for either side in that referendum.
The policy has been criticised by some for being too confusing.
Home secretary Priti Patel criticised Labour for its refusal to rule out keeping freedom of movement with the EU.
Read more: Conservatives refuse to reveal cost of spending pledges
“For the second time in a week, Corbyn’s Labour have confirmed they want to see uncontrolled and unlimited immigration, whatever the results of the two chaotic referendums they want next year,” she said.
“The choice at this election is between a Conservative majority government that will get Brexit done and deliver on the people’s priorities like controlling immigration, or a weak coalition headed by Corbyn in a gridlocked parliament that refuses to control immigration.”


A task in Flink is the basic unit of execution. It is the place where each parallel instance of an operator is executed As an example, an operator with a parallelism of 5 will have each of its instances executed by a separate task.
The StreamTask
is the base for all different task sub-types in Flink’s streaming engine. This document goes through
the different phases in the lifecycle of the StreamTask
and describes the main methods representing each of these
phases.
Because the task is the entity that executes a parallel instance of an operator, its lifecycle is tightly integrated
with that of an operator. So, we will briefly mention the basic methods representing the lifecycle of an operator before
diving into those of the StreamTask
itself. The list is presented below in the order that each of the methods is called.
Given that an operator can have a user-defined function (UDF), below each of the operator methods we also present
(indented) the methods in the lifecycle of the UDF that it calls. These methods are available if your operator extends
the AbstractUdfStreamOperator
, which is the basic class for all operators that execute UDFs.
// initialization phase
OPERATOR::setup
UDF::setRuntimeContext
OPERATOR::initializeState
OPERATOR::open
UDF::open
// processing phase (called on every element/watermark)
OPERATOR::processElement
UDF::run
OPERATOR::processWatermark
// checkpointing phase (called asynchronously on every checkpoint)
OPERATOR::snapshotState
// termination phase
OPERATOR::close
UDF::close
OPERATOR::dispose
In a nutshell, the setup()
is called to initialize some operator-specific machinery, such as its RuntimeContext
and
its metric collection data-structures. After this, the initializeState()
gives an operator its initial state, and the
open()
method executes any operator-specific initialization, such as opening the user-defined function in the case of
the AbstractUdfStreamOperator
.
Attention The initializeState()
contains both the logic for initializing the
state of the operator during its initial execution (e.g. register any keyed state), and also the logic to retrieve its
state from a checkpoint after a failure. More about this on the rest of this page.
Now that everything is set, the operator is ready to process incoming data. Incoming elements can be one of the following:
input elements, watermark, and checkpoint barriers. Each one of them has a special element for handling it. Elements are
processed by the processElement()
method, watermarks by the processWatermark()
, and checkpoint barriers trigger a
checkpoint which invokes (asynchronously) the snapshotState()
method, which we describe below. For each incoming element,
depending on its type one of the aforementioned methods is called. Note that the processElement()
is also the place
where the UDF’s logic is invoked, e.g. the map()
method of your MapFunction
.
Finally, in the case of a normal, fault-free termination of the operator (e.g. if the stream is finite and its end is
reached), the close()
method is called to perform any final bookkeeping action required by the operator’s logic (e.g.
close any connections or I/O streams opened during the operator’s execution), and the dispose()
is called after that
to free any resources held by the operator (e.g. native memory held by the operator’s data).
In the case of a termination due to a failure or due to manual cancellation, the execution jumps directly to the dispose()
and skips any intermediate phases between the phase the operator was in when the failure happened and the dispose()
.
Checkpoints: The snapshotState()
method of the operator is called asynchronously to the rest of the methods described
above whenever a checkpoint barrier is received. Checkpoints are performed during the processing phase, i.e. after the
operator is opened and before it is closed. The responsibility of this method is to store the current state of the operator
to the specified state backend from where it will be retrieved when
the job resumes execution after a failure. Below we include a brief description of Flink’s checkpointing mechanism,
and for a more detailed discussion on the principles around checkpointing in Flink please read the corresponding documentation:
Data Streaming Fault Tolerance.
Following that brief introduction on the operator’s main phases, this section describes in more detail how a task calls
the respective methods during its execution on a cluster. The sequence of the phases described here is mainly included
in the invoke()
method of the StreamTask
class. The remainder of this document is split into two subsections, one
describing the phases during a regular, fault-free execution of a task (see Normal Execution), and
(a shorter) one describing the different sequence followed in case the task is cancelled (see Interrupted Execution),
either manually, or due some other reason, e.g. an exception thrown during execution.
The steps a task goes through when executed until completion without being interrupted are illustrated below:
TASK::setInitialState
TASK::invoke
create basic utils (config, etc) and load the chain of operators
setup-operators
task-specific-init
initialize-operator-states
open-operators
run
close-operators
dispose-operators
task-specific-cleanup
common-cleanup
As shown above, after recovering the task configuration and initializing some important runtime parameters, the very
first step for the task is to retrieve its initial, task-wide state. This is done in the setInitialState()
, and it is
particularly important in two cases:
If it is the first time the task is executed, the initial task state is empty.
After recovering any initial state, the task goes into its invoke()
method. There, it first initializes the operators
involved in the local computation by calling the setup()
method of each one of them and then performs its task-specific
initialization by calling the local init()
method. By task-specific, we mean that depending on the type of the task
(SourceTask
, OneInputStreamTask
or TwoInputStreamTask
, etc), this step may differ, but in any case, here is where
the necessary task-wide resources are acquired. As an example, the OneInputStreamTask
, which represents a task that
expects to have a single input stream, initializes the connection(s) to the location(s) of the different partitions of
the input stream that are relevant to the local task.
Having acquired the necessary resources, it is time for the different operators and user-defined functions to acquire
their individual state from the task-wide state retrieved above. This is done in the initializeState()
method, which
calls the initializeState()
of each individual operator. This method should be overridden by every stateful operator
and should contain the state initialization logic, both for the first time a job is executed, and also for the case when
the task recovers from a failure or when using a savepoint.
Now that all operators in the task have been initialized, the open()
method of each individual operator is called by
the openAllOperators()
method of the StreamTask
. This method performs all the operational initialization,
such as registering any retrieved timers with the timer service. A single task may be executing multiple operators with one
consuming the output of its predecessor. In this case, the open()
method is called from the last operator, i.e. the
one whose output is also the output of the task itself, to the first. This is done so that when the first operator starts
processing the task’s input, all downstream operators are ready to receive its output.
Attention Consecutive operators in a task are opened from the last to the first.
Now the task can resume execution and operators can start processing fresh input data. This is the place where the
task-specific run()
method is called. This method will run until either there is no more input data (finite stream),
or the task is cancelled (manually or not). Here is where the operator specific processElement()
and processWatermark()
methods are called.
In the case of running till completion, i.e. there is no more input data to process, after exiting from the run()
method, the task enters its shutdown process. Initially, the timer service stops registering any new timers (e.g. from
fired timers that are being executed), clears all not-yet-started timers, and awaits the completion of currently
executing timers. Then the closeAllOperators()
tries to gracefully close the operators involved in the computation by
calling the close()
method of each operator. Then, any buffered output data is flushed so that they can be processed
by the downstream tasks, and finally the task tries to clear all the resources held by the operators by calling the
dispose()
method of each one. When opening the different operators, we mentioned that the order is from the
last to the first. Closing happens in the opposite manner, from first to last.
Attention Consecutive operators in a task are closed from the first to the last.
Finally, when all operators have been closed and all their resources freed, the task shuts down its timer service, performs its task-specific cleanup, e.g. cleans all its internal buffers, and then performs its generic task clean up which consists of closing all its output channels and cleaning any output buffers.
Checkpoints: Previously we saw that during initializeState()
, and in case of recovering from a failure, the task
and all its operators and functions retrieve the state that was persisted to stable storage during the last successful
checkpoint before the failure. Checkpoints in Flink are performed periodically based on a user-specified interval, and
are performed by a different thread than that of the main task thread. That’s why they are not included in the main
phases of the task lifecycle. In a nutshell, special elements called CheckpointBarriers
are injected periodically by
the source tasks of a job in the stream of input data, and travel with the actual data from source to sink. A source
task injects these barriers after it is in running mode, and assuming that the CheckpointCoordinator
is also running.
Whenever a task receives such a barrier, it schedules a task to be performed by the checkpoint thread, which calls the
snapshotState()
of the operators in the task. Input data can still be received by the task while the checkpoint is
being performed, but the data is buffered and only processed and emitted downstream after the checkpoint is successfully
completed.
In the previous sections we described the lifecycle of a task that runs till completion. In case the task is cancelled at any point, then the normal execution is interrupted and the only operations performed from that point on are the timer service shutdown, the task-specific cleanup, the disposal of the operators, and the general task cleanup, as described above.


{% for address in site.address %} {{ address.line }} {% endfor %}
Copyright © {{ site.copyright }} 20{{ site.time | date: '%y' }}


Installation¶
At the command line with pip:
$ pip install lenstronomy
Or, if you have virtualenvwrapper installed:
$ mkvirtualenv lenstronomy
$ pip install lenstronomy
You can also clone the github repository for development purposes.
Requirements¶
Make sure the standard python libraries as specified in the requirements. The standard usage does not require all libraries to be installed, in particular the different posterior samplers are only required when being used.
In the following, a few specific cases are mentioned that may require special attention in the installation and settings, in particular when it comes to MPI and HPC applications.
MPI¶
MPI support is provided for several sampling techniques for parallel computing. A specific version of the library schwimmbad is required for the correct support of the moving of the likelihood elements from one processor to another with MPI. Pay attention ot the requirements.
NUMBA¶
Just-in-time (jit) compilation with numba can provide significant speed-up for certain calculations. There are specific settings for the settings provided as per default, but these may need to be adopted when running on a HPC cluster. You can define your own configuration file in your $XDG_CONFIG_HOME/lenstronomy/config.yaml file. E.g. (check your system for the path):
$ ~/.conf/lenstronomy/config.yaml
following the format of the default configuration which is here.
FASTELL¶
The fastell4py package, originally from Barkana (fastell), is required to run the PEMD (power-law elliptical mass distribution) lens model and can be cloned from: https://github.com/sibirrer/fastell4py (needs a fortran compiler). We recommend using the EPL model as it is a pure python version of the same profile.
$ sudo apt-get install gfortran
$ git clone https://github.com/sibirrer/fastell4py.git <desired location>
$ cd <desired location>
$ python setup.py install --user
Check installation by running tests¶
You can check your installation with pytest:
$ cd <lenstronomy_repo>
$ py.test
Or you can run a partial test with:
$ cd <lenstronomy_repo>
$ py.test/test/test_LensModel/
You can also run the tests with tox in a virtual environment with:
$ cd <lenstronomy_repo>
$ tox
Note: tox might have trouble with the PyMultiNest installation and the cmake part of it.


By clicking on the button you will start to subscribe to a timer service
subscribe to service1Service is not available yet
subscribe to service2Service is not available yet
subscribe to service3Service is not available yet
subscribe to service4Service is not available yet
subscribe to service5Service is not available yet


The
utility prints out the mail header lines from the invoker's mailbox.
The following options are available:
Just print a count of messages and exit.
The supplied
is examined instead of the invoker's mailbox. If the
option is used, the
argument should not be used. Read from standard input if file name
is given.
Only mail from addresses containing the supplied string are printed.
If
is given, the
mailbox is examined instead of the invoker's own mailbox. (Privileges are required.)
If set, the location of the invoker's mailbox. Otherwise, the default in
is used.
The
command appeared in


WITNESS LIST Human Services Committee April 6, 1999-2:00P HB 689 For: Rich, Sid (Texas Assoc. of Residential Care Communities) St. Clair, Johnny (Unlimited Care at Waco) On: Hayes, Aaryce (Advocacy Inc.) HB 691 For: Rich, Sid (Texas Assoc. of Residential Care Communities) St. Clair, Johnny (Unlimited Care at Waco) Against: Mathews, CW "Mat" (Texas Organization of Residential Care Homes) On: Gold, Marc (DHS) Hayes, Aaryce (Advocacy Inc.) HB 2017 On: Zambrzycki, Sharon (DHS) HB 3341 For: Denton, Ann (The Enterprise Foundation) Gantt, Melanie (Mental Health Assoc. in Texas) Halligan, Mike (Texas Mental Health Consumers) Harris, Lauralee (Mental Health Assoc. of Tarrant County) Hayes, Aaryce (Advocacy Inc.) Siddons, Bonnie (Mental Health Housing Development Corp.) HB 3386 For: Marks, David (Self) Speights, Sara (Tx. Assoc. for Home Care) On: Hayes, Aaryce (Advocacy Inc.) Mixson, Paula (DPRS) HB 3638 For: Bower, Bruce (Tx. Legal Services Center) Comstock-Galagan, Jim (Advocacy Inc) HB 3639 For: Bower, Bruce (Tx. Legal Services Center) Bresette, Patrick (Center for Public Policy Priorities) Carey, Kay (Assoc. of Substance Abuse Programs) Cole, Pat (National Training Center on Domestic & Sexual Violence) Greer, Patricia (Tx. Assoc. of Addiction Professionals) Hernandez, Leslie (National Assoc. of Social Workers) Martin, Vanessa (Addiction Action Alliance) Murphree, Susan (Advocacy Inc.) HB 3729 On: Dunkelberg, Anne (Center for Public Policy Priorities) HB 3741 For: Villarreal, Richard (Self) HB 3778 For: Hernandez, Leslie (NASW) On: Webster, Sarah (DPRS)


Algorithm and scientific questions: satijalab.org
Module wrapping issues: Ted Liefeld < jliefeld at cloud dot ucsd dot edu>
Seurat is an R package designed for QC, analysis, and exploration of single-cell RNA-seq data. Seurat aims to enable users to identify and interpret sources of heterogeneity from single-cell transcriptomic measurements, and to integrate diverse types of single-cell data.
Within GenePattern, Seurat is accessed as several different modules exposing different portions of the Seurat workflow; Seurat.Preprocessing, Seurat.BatchCorrection, Seurat.QC and Seurat.Clustering. This module, Seurat.Clustering performs UMAP clustering and marker identification on single-cell RNA-Seq data.
The Seurat.Clustering module performs uses Seurat version 3.0.2. It performs following steps from Seurat to the input dataset;
GenePattern 3.9.11 or later (dockerized).
| Name | Description |
|---|---|
| input seurat rds file* | A rds file containig a Seurat object. |
| output filename* | The output filename prefix used for all output files. |
| maximum dimension | The maximum number of clusters to attempt to find. |
| resolution* | The resolution to use to find clusters. |
| reduction* | The reduction to use (UMAP). Other reductions (e.g. TSNE) may be added later. |
| seed* | The random number seed to use. |
* - required


OS simulator
Home
Wiki
Syncronisation
Producer-consumer
Dining Philosopher
Barrier Semaphore
Reader-Writer
Sleeping Barber