Most disk-operating systems provide minimal file functionality. A file is simply treated as a sequence of bytes; applications can copy data from files to memory or vice versa, but nothing more elaborate. GEOS provides much more elaborate functionality with its
Virtual Memory
files.
Virtual Memory (VM) is very useful for two reasons. First, it is often impractical to read an entire disk file into contiguous memory; indeed, it is impossible to do so if the file is larger than 64K. The use of virtual memory allows each file to grow arbitrarily large. Second, each disk file is one long, cumbersome stream of data. By using virtual memory files, applications can break files down into smaller, more manageable blocks which the memory manager can handle more efficiently.
1 Design Philosophy
2 VM Structure
2.1 The VM Manager
2.2 VM Handles
2.3 Virtual Memory Blocks
2.4 VM File Attributes
3 Using Virtual Memory
3.1 How to Use VM
3.2 Opening or Creating a VM File
3.3 Changing VM File Attributes
3.4 Creating and Freeing Blocks
3.5 Attaching Memory Blocks
3.6 Accessing and Altering VM Blocks
3.7 VM Block Information
3.8 Updating and Saving Files
3.9 Closing Files
3.10 The VM File's Map Block
3.11 File-Access Synchronization
3.12 Other VM Utilities
4 VM Chains
4.1 Structure of a VM Chain
4.2 VM Chain Utilities
5 Huge Arrays
5.1 Structure of a Huge Array
5.2 Basic Huge Array Routines
5.3 Huge Array Utilities
The GEOS Virtual Memory file management system is designed to provide features not easily available with standard file systems. Designed primarily for use by applications for long-term data storage, it is also used by the system for many other purposes, such as to provide state saving (in the UI). The main benefits it provides to applications include the following:
Virtual memory can be thought of as a heap stored in a disk file. Like the global heap, it is divided into blocks which are 64K or smaller; each block is accessed via a handle. Blocks can be locked, which brings them into main memory. If blocks are modified while in memory, they are copied back to the file when the file is updated.
The primary component of virtual memory is the VM file. The VM file consists of a VM File Header, a collection of VM blocks, and a special structure called a VM Header. The VM File Header is a standard GEOS file header, containing the file's extended attributes and system bookkeeping information. Geodes may not access it directly; instead, they can make calls to the extended attributes routines to access data in the header. The VM blocks and the VM Header do not occupy fixed places in the file. In particular, the VM Header does not necessarily come before all the blocks. Instead, the VM File Header stores the offset within the file to the VM Header, and the VM Header stores information about the blocks (such as their locations in the file). Furthermore, the blocks in a VM file are arranged in no particular order; they are not necessarily arranged in the order they were created, or in any other sequence.
The VM Header maintains all the bookkeeping information about the VM file. For example, it contains the VM Block Table. The block table is much like the global handle table. Block handles are offsets into the block handle table; a block's entry in the table contains information about the block, such as the block's location in the file. Usually, the Block Table contains entries for blocks that haven't been created yet; when all of these handles have been used, the VM Manager expands the block table. For details, see Virtual Memory Blocks .
The VM manager can be thought of as a memory manager for a disk-based heap, providing all the services a memory manager would and more. The VM manager provides for allocation, use, and deallocation of virtual memory blocks. It manages the Block Table, enlarging it as necessary when more VM block handles are needed. It keeps track of which VM blocks have been loaded into memory, which are currently in use, and which have been "dirtied" (modified) since they were read from the disk; it also keeps track of how many threads are accessing a given VM file at any time. The VM manager also accomplishes all swapping and read/write operations involving VM files.
There are several different types of handles associated with VM files. It is important not to get them confused.
Most file systems treat files as a string of bytes. A user can read a file sequentially or can start reading at a specified distance into the file. This makes it difficult to work with large files. Reading an entire large file into memory may be impractical, and it may be difficult to access different parts of the file at once.
For this reason, GEOS divides its Virtual Memory files into VM blocks. This division is entirely internal to GEOS. When the file is written to disk, it is still a sequence of bytes, and other operating systems can copy the file normally. However, GEOS geodes can access the file much more conveniently by specifying the blocks they wish to access.
A VM block is a sequence of bytes in a VM file. It must be small enough to fit in a global memory block (i.e. 64K or less, preferably 2K-6K). It may move about in the file; for this reason, it is accessed by a VM block handle. Blocks are either free or used . A used block has an entry in the block table and also a space allocated in the file. (This could be a block of free space, which will be freed the next time the file is compacted.) A free block has a slot in the file's handle table but no space in the file; it is available if a thread needs to allocate a block.
Blocks persist after a file has been closed and GEOS has been shut down. A given block is always accessed by the same block handle. There are utilities to copy blocks within a VM file or between files. Blocks in a VM file are in no particular order. If an application wants to set up a sequence of blocks, it can create a VM Chain, in which the first word of each block is the handle of the next block in the chain. However, even chained blocks will probably not be in order in the actual file.
Each VM block can be assigned a "user ID" number. You can request the handles of the VM blocks with any given ID number. You do not have to assign ID numbers to blocks, but it is sometimes convenient. The ID numbers are stored in the handles' entries in the block table, not in the blocks themselves; this makes it easy to find a block with a specified user ID. User IDs can be changed at will with the routine
VMModifyUserID()
. Note that all user IDs from 0xff00 to 0xffff are reserved for system use. You can find a block with a specific user ID by calling
VMFind()
; see You can locate VM blocks by their user ID numbers. The routine VMFind() takes three arguments: the VM file handle, a VM block handle, and the user ID for which to look. The routine looks through the block table, starting with the handle after th
.
There are two ways you can create a VM block. The first is to request a VM block: You specify the size of the block, the block is created, and you are returned the handle. This is the method ordinarily used. The second method is to attach memory to a block: You create a global memory block, and instruct the VM manager to add it to the VM file. There are sometimes advantages to this technique; for example, you can create an LMem heap, and then attach it to the VM file; it will then be saved with the file. You can also attach a memory block to an existing VM block; this will destroy whatever used to be in the VM block and replace it with the contents of the new block.
You can dynamically resize a VM block by locking it into memory, resizing the memory block, and saving it back to the disk.
These techniques are described in detail in Creating and Freeing Blocks and Attaching Memory Blocks .
When a VM block is freed, the VM manager will note that there is empty space in the file. It will use that space when new blocks are allocated. However, since new blocks will probably not fit exactly in the spaces left by freed blocks, some space may be wasted.
In time, the percentage of wasted file space can grow unacceptably large. To prevent this, the Virtual Memory manager periodically compacts the files. When the ratio of data to free space drops below a certain threshold, the Virtual Memory manager copies the data in the file over the free space. While a file is being compacted, any requests for access to the file will block until compaction is finished. Note that the format of the data is not changed; the free space between data blocks is simply removed.
When a geode creates a VM file, it can specify a "compaction threshold." When the percentage of used space drops below this threshold, the VM manager will automatically compact the file without any attention from the application. The geode should take care in setting the threshold. If the threshold is too low, the file may grow unacceptably large before it is compacted; on the other hand, if the threshold is too high, the VM manager might spend too much time compacting the file for relatively low gains. The application can specify a threshold of zero; this will cause the system default threshold to be used.
Note that if a file is in "backup mode," the file will be compacted only on calls to
VMSave()
,
VMSaveAs()
, or
VMRevert()
. If the file is not in backup mode, it can be compacted on any call to
VMUpdate()
.
When a block is locked into memory, the VM manager copies the data from the disk block to the global memory block. When the block is unlocked, the VM manager assumes that it can discard the memory block, since the data is already on the disk in the VM block.
If you alter the data in a block, you must notify the VM manager of this fact. You do this by marking the block as
dirty
. When a block has been marked dirty, the VM manager knows that the version in memory is more up-to-date than the version in the disk file. If the flag VMA_SYNC_UPDATE is
off
(the default), the block will be written back to the file as soon as possible. If the attribute is
on
, the block will not be copied back to the disk file until
VMUpdate()
is called; until then, the block will be copied to the disk swap space if memory is needed. The next time you lock the block, you will be given the new, changed version.
When you want to write the dirty blocks to the disk, you can instruct the VM manager to update the file. This copies all the dirtied blocks back to the disk and marks all blocks as clean . The VM manager also automatically updates the file when it is closed. The updating routines check if the file is currently clean; thus, very little time is lost if you try to update a clean file.
The VM manager can be instructed to notify all users of a file when the file changes from clean to dirty. This has two main uses: it helps maintain data synchronization if many geodes are using the same file, and it lets the document control objects know when to enable the "Save" and "Revert" triggers. (See the Documents Objects chapter.)
The VM manager can be instructed to maintain a backup of a file. If it is so instructed, it will not overwrite the original block when it updates it; instead, it will keep a copy of the original block as well as a copy of the new version. This is transparent to the application. When the VM Manager is instructed to save the file, it deletes all the backup blocks. If it is instructed to revert the file to its last-saved version, it replaces the new version of the changed blocks with the backup versions, thus restoring the file to its condition as of the last time it was saved. If the VM manager is instructed to "save-as" the file, it will create a new file, which does not contain the backup blocks; that is, it contains the file as it currently exists. It will then revert the original file to its last-saved version and close it.
VMAttributes
Each VM file has a set of attributes which determine how the VM Manager treats the file. These attributes are specified by a set of
VMAttributes
flags. When a VM file is created, all of these attributes are off; after a file has been created, you can change the attributes with
VMSetAttributes()
(see Changing VM File Attributes
). The following flags are available:
VMUpdate()
,
VMSave()
, etc.). This attribute is
off
by default (indicating that the VM manager should feel free to update blocks whenever they are unlocked). You should set this attribute if the file might not be in a consistent state every time a block is unlocked.
MSG_META_VM_FILE_DIRTY
to each process that has the file open. (This message is defined for
MetaClass
, so any object can handle it.)
OLMBH_inUseCount
is non-zero. This attribute must be set if the file contains object blocks. If this attribute is set, each object block will be kept in memory as long as any thread is using an object in the block.
CompactObjBlock()
. This allows a VM file to contain generic object blocks.
VMLock()
. This attribute must be set if the VM file contains object blocks.The most common way applications will use Virtual Memory is for their data files. However, there are other uses; for example, VM provides a convenient way to maintain working memory. Some data structures (such as Huge Arrays) can only exist in VM files; an application may create a temporary VM file just for these structures.
Accessing and Altering VM Blocks
There are five basic steps to using VM files. The steps are outlined below, and described in greater detail in the following sections.
VMOpen()
. This routine can be used to open an existing VM file or create a new one. If you use the document control objects, they will open and create files automatically. Once you have created a VM file, you may want to change its attributes with
VMSetAttributes()
.
VMLock()
. You can also create new blocks with
VMAlloc()
and
VMAttach()
.
VMUnlock()
. If you change the memory, you should mark it dirty with
VMDirty()
before unlocking it. This ensures that the new version of the block will be written to the disk.
VMClose()
when you are done with the file. It will update the file and close it.VMOpen(), VMOpenType, VMAccessFlags
To create or open a VM file, call the routine
VMOpen()
. You may not need to open and create files directly; if you use the document control objects, they automatically create and open files as the user requests. (See the Document Objects chapter.) VMOpen()
looks for the file in the thread's working directory (unless a temporary file is being created, as described below).
VMOpen()
takes four arguments and returns the file handle. The arguments are:
VMAccessFlags
This argument is a set of flags which specifies how the file is accessed. The flags are described below.
VMOpenType
enumerated type
VMOpenType
are described below.
When you use
VMOpen()
, you must specify how the file should be opened.
You do this by passing a member of the
VMOpenType
enumerated type. The types are as follows:
VMOpen()
will return an error condition.
VMOpenType
values with a bit-wise
or
.
You also have to specify what type of access to the file you would like. You do this by passing a record of
VMAccessFlags
. This is a byte-length bitfield. The following flags are available:
If you open a file with VMAF_FORCE_READ_ONLY, it's generally a good idea to also open it with VMAF_FORCE_DENY_WRITE. When you open a file VMAF_FORCE_READ_ONLY, if the file is writable, and is located on a writable device which can be used by other machines (e.g. a network drive), the kernel will load the entire file into memory and make the blocks non-discardable (even when they are clean); this keeps the file you see consistent, even if another user changes the version of the file on the disk. However, this can cause problems if the machine has limited swap space. If the file is opened with VMAF_FORCE_DENY_WRITE, no other device will be allowed to change the file while you have it open, which means the kernel can just load and discard blocks as necessary.
The routine
VMOpen()
returns the file handle. If it cannot satisfy the request, it returns a null handle and sets the thread error word. The error word can be recovered with the
ThreadGetError()
routine. The possible error conditions are:
VMOpen()
was passed VMO_CREATE_ONLY, but the file already exists.
VMOpen()
was passed VMO_OPEN, but the file does not exist.
VMOpen()
was instructed to open an invalid VM file (or a non-VM file).
VMOpen()
cannot create the file (but it does not already exist).
VMOpen()
was passed VMO_CREATE_TRUNCATE; the file exists but could not be truncated.
VMOpen()
was passed VMAF_FORCE_READ_WRITE, but the file or disk was write-protected.VMGetAttributes(), VMSetAttributes()
When a VM file is created, it is given a set of
VMAttributes
(see VMA_SYNC_UPDATE Allow synchronous updates only. Instructs VM Manager to update the file only when you call un updating routine (VMUpdate(), VMSave(), etc.). This attribute is off by default (indicating that the VM manager should feel free to upd
).
These attributes can be examined with the routine
VMGetAttributes()
. The routine takes one argument, namely the handle of the VM file (which is overridden if a default VM file is set). It returns the
VMAttributes
flags.
You can change the attributes by calling
VMSetAttributes()
. This routine takes three arguments: the file handle (which may be overridden), a set of bits which should be turned on, and a set of bits which should be turned off. It returns the new
VMAttributes
flags.
VMAlloc(), VMAllocLMem(), VMFree()
Once you have created a VM file, you have to allocate blocks in order to write data to the file.
The usual way to do this is with
VMAlloc()
. This routine takes three word-sized arguments:
VMFind()
(see You can locate VM blocks by their user ID numbers. The routine VMFind() takes three arguments: the VM file handle, a VM block handle, and the user ID for which to look. The routine looks through the block table, starting with the handle after th
).
VMAttach()
(see Attaching Memory Blocks
).
The routine returns the handle of the VM block. Before you can use the block, you have to lock it with
VMLock()
. The block is marked dirty when it is allocated.
There is a routine to allocate a block and initialize it as an LMem heap. This is useful if you are storing object blocks in a VM file. The routine,
VMAllocLMem()
, takes three arguments:
LMemTypes
enumerated typeThe routine creates a VM block and allocates a global memory block to go with it. It initializes the heap in the global block and marks the block as dirty. The LMem heap will begin with two LMem handles and a 64-byte heap; this will grow as necessary. The VM block will have a user ID of zero; you can change this if you wish. The routine returns the handle of the new VM block.
There are two other ways to create LMem blocks in a VM file; these ways give you more control of the block's initialization. You can allocate a VM block normally, lock that block, then get the handle of the associated memory block and initialize an LMem heap in it; or you can allocate an LMem heap normally, and attach that memory block to the VM file using
VMAttach()
. For more details on LMem heaps, see the Local Memory chapter.
To free a VM block, call
VMFree()
. This routine is passed two arguments: the VM file handle, and the VM block handle. The handle will immediately be freed, even if it is locked. Any associated memory will also be freed. If you want to keep the memory, detach the global memory block from the file (with
VMDetach()
) before you free the block.
VMAttach(), VMDetach()
When you use
VMAlloc()
, the VM manager allocates a global memory block and attaches it to a VM block. However, sometimes you want to allocate the block yourself, or you may have an existing memory block which you want to copy into the VM file.
To do this, you call the routine
VMAttach()
.
VMAttach()
takes three arguments:
VMAttach()
attaches the global memory block to the VM block. The VM Manager becomes the owner of the memory block. The next time the file is updated, the memory block will be copied to the file.
VMAttach()
returns the handle of the VM block. If it could not perform the attach, it returns a null handle and leaves the global memory block unchanged.
You can also detach the global memory block from a VM block. The routine
VMDetach()
disassociates a global memory block from its VM block. The routine takes three arguments: the VM file handle; the VM block handle; and the
GeodeHandle
of the geode which will be made the owner of the memory block. (Passing a null
GeodeHandle
will make the calling geode the block's owner.) The VM manager disassociates the memory block from the VM block, changes the memory block's owner, marks it "non-discardable," and returns its handle. If the VM block is not currently in memory,
VMDetach()
will automatically allocate a memory block, copy the VM block's data into it, and return the memory block's handle. If the VM block was dirty, the block will be updated to the file before it is detached. The next time the VM block is locked, a new global memory block will be allocated for it.
VMLock(), VMUnlock(), VMDirty(), VMFind(), VMModifyUserID(), VMPreserveBlocksHandle()
Once you have opened a VM file and allocated blocks, you will need to access blocks. The VM library provides many routines for doing this.
If you need to access the data in a VM file, you can use the routine
VMLock()
. This routine moves a VM block onto the global heap. It does this by allocating a global memory block (if the VM block is not already associated with a memory block), reallocating the global block if it had been discarded, locking the memory block on the global heap, and copying the VM block into the global block, if necessary. (It will copy the VM block to memory only if necessary, i.e. if the memory block is newly-allocated, or had been discarded and reallocated.)
VMLock()
takes three arguments: the handle of the VM file, the VMBlockHandle of the block to lock, and a pointer to a memHandle variable.
It returns a pointer to the start of the block, and writes the global block's handle into the memHandle variable. You can now access the block the same way you would any other block, with one exception: When you are done with the block, you do not call
MemUnlock()
; instead, call the routine
VMUnlock()
, passing it the handle of the
global memory
block (not the handle of the VM block). This will unlock the global block on the heap.
If you alter the global memory block, you will need to notify the VM manager of this so it will know to copy the changes back to the VM file.
You do this by calling the routine
VMDirty()
.
VMDirty()
takes one argument, the handle of the global memory block (
not
the VM block). It is important to dirty the block while it is still locked on the heap; as soon as you unlock a clean block, the VM manager may choose to discard it. Dirty blocks are copied back to the VM file when it is updated. Note that if an object in a VM block is marked dirty (via
ObjMarkDirty()
), the block is automatically dirtied. Similarly, if you attach a global memory block to a VM block (via
VMAttach()
), the VM block is automatically dirtied.
You can dynamically resize VM blocks.
To do this, lock the VM block with
VMLock()
; then resize the global memory block with
MemReAlloc()
. Be sure to mark the block dirty so the changes will be copied to the disk file. Note that although the global memory block will remain locked, it may move on the global heap when it is resized. You will therefore need to dereference the global memory handle (with
MemDeref()
) before accessing the memory.
You can locate VM blocks by their user ID numbers.
The routine
VMFind()
takes three arguments: the VM file handle, a VM block handle, and the user ID for which to look. The routine looks through the block table, starting with the handle
after
the one passed, until it finds a block with the specified user ID. If it does not find such a block, it returns a null handle; otherwise, it returns the block's VMBlockHandle. Thus, by passing in a block handle of zero, you will get the handle of the first block with the specified ID; by passing back in that block's handle, you will get the next block with that ID; and so on, until you get all the blocks (after which you will be returned a null handle).
You can change a block's user ID number by calling the routine
VMModifyUserID()
. This routine takes three arguments: the VM file handle, the VM block handle, and the new user ID number. Since user IDs are maintained in the block table, not in the blocks themselves, it doesn't matter whether the block is locked, or even whether it is associated with data in the file. (For example, a block allocated with a size of zero can have its user-ID changed.)
Ordinarily, the VM manager can free any unlocked, clean global block if the space is needed. However, you can instruct the VM manager not to free the global block associated with a specific block by calling the routine
VMPreserveBlocksHandle()
. The routine takes two arguments, namely the VM file handle and the VM block handle. It sees to it that the specified VM block will remain attached to the same global block until the VM block is specifically detached (or reattached).
VMVMBlockToMemBlock(), VMMemBlockToVMBlock(), VMInfo()
Several utilities are provided to give you information about VM blocks.
If you know the handle of a VM block, you can find out the handle of the associated global block by calling the routine
VMVMBlockToMemBlock()
. This routine takes two arguments, namely the VM file handle and the VM block handle. It returns the global memory handle of the associated block; however, note the caveats regarding global handles in the above section. If the VM block is not currently associated with a global memory block, the routine will allocate a memory block, copy the VM block into it, and return its handle. If the VM handle is not associated with any data in the file and is not attached to a global memory block,
VMVMBlockToMemBlock()
returns a null handle.
Conversely, if you know the handle of a global memory block and want to find out the VM file and block to which it is attached, call the routine
VMMemBlockToVMBlock()
. This routine takes two arguments: the global memory handle, and a pointer to a
VMFileHandle
variable. It returns the VM block handle of the associated VM block, and writes the handle of the VM file to the address passed. If the global memory block is not attached to a VM file, it returns null handles.
The Boolean routine
VMInfo()
is an omnibus information routine. It takes three arguments: the handle of a VM file, the handle of a VM block, and a pointer to a
VMInfoStruct
structure (described below in VMInfoStruct
). If the VM block is free, out of range, or otherwise invalid, it returns
false
; otherwise, it returns
true
(i.e. non-zero) and fills in the fields of the
VMInfoStruct
.
Code Display 18-1 VMInfoStruct
/* This is the definition of the VMInfoStruct. A pointer to a VMInfoStruct is * passed to the routine VMInfo(). The routine fills in the structure's fields. */ typedef struct { MemHandle mh; /* Null handle returned if no block is attached */ word size; /* Size of VM block in bytes */ word userID; /* User ID (or zero if no user ID was specified) */ } VMInfoStruct;
VMUpdate(), VMSave(), VMSaveAs(), VMRevert(), VMGetDirtyState() VMSave()
When you dirty a memory block, that action notifies the VM manager that the block will need to be written back to the file. If the attribute VMA_SYNC_UPDATE is off , the VM manager will try to update the block to the disk file as soon as the block is unlocked, and will then mark the block as clean . However, if the flag is on , the manager does not write the block until it is specifically told to update the file.  If you use the document control objects, they will take care of updating and saving the file. However, you may need to call the updating routines specifically.
The routine
VMUpdate()
instructs the VM manager to write all dirty blocks to the disk. It takes one argument, the VM file handle (which is overridden if a thread file has been set). It returns zero if the update proceeded normally; otherwise, it returns either one of the
FileErrors
or one of the three
VMUpdate()
status codes:
VMUpdate()
is optimized for updating clean files; thus, it costs very little time to call
VMUpdate()
when you are not sure if the file is dirty. If a file is auto-saved,
VMUpdate()
is used.
A VM file can maintain backup copies of updated blocks. If so, updating the file will write changes to the disk, but will not alter those backup blocks. To finalize the changes, call the routine
VMSave()
. This routine updates the file, then deletes all the backup blocks and compacts the file. If the file does not have backup capability,
VMSave()
acts the same as
VMUpdate()
.
If a file has the backup capability, you cannot directly access the backup blocks. However, you can instruct the VM manager to restore the file to its last-saved state. The command
VMRevert()
causes the VM manager to check the VM file for blocks which have backups. It then deletes the
non-backup
block, and changes the backup block into a regular block. It also discards all blocks in memory that were attached to the blocks which just reverted. The file will then be in its condition as of the last time it was saved. The routine may not be used on files which do not have the flag VMA_BACKUP set.
You can save a file under a new name with the routine
VMSaveAs()
. If the file has backup capability, the old file will be restored to its last-saved condition (as if
VMRevert()
had been called); otherwise, the old file will be left in the file's current state. The routine is passed the name of the new file.
VMSaveAs()
copies all the blocks from the old file to the new one. If a block has a backup copy, the more recent version is copied. The new file will thus have the file in its current state; block handles will be preserved. After the new file has been created, if the file has backup-capability,
VMSaveAs()
reverts the original file to its last-saved state. It then closes the old file and returns the handle of the new file.
If you manage VM files with the document control objects, you generally don't have to call the update or save routines. The document control objects will set up a file menu with appropriate commands ("Save," "Save As," etc.), and will call the appropriate routines whenever the user chooses a command.
If you need to find out whether a file is dirty, call the routine
VMGetDirtyState()
. This routine returns a two-byte value. The more significant byte is non-zero if any blocks have been dirtied since the last update or auto-save. The less significant byte is non-zero if any blocks have been dirtied since the last save, save-as, or revert action. If the file does not have backup-capability, both bytes will always be equal. Note that
VMUpdate()
is optimized for clean files, so it is generally faster to call
VMUpdate()
even if the file might be clean, rather than checking the dirty-state with
VMGetDirtyState()
.
VMClose()
When you are done with a VM file for the time being, you should close it with
VMClose()
. This routine updates all the dirty blocks, frees all the global memory blocks attached to the file, and closes the file (thus freeing its handle). The routine is passed two arguments. The first is the handle of the file to close. The second is a flag,
noErrorFlag
. If this flag is FILE_NO_ERROR,
VMClose()
will not return error conditions; if it could not successfully close the file, it will fatal-error.
If
noErrorFlag
is
false
(i.e., zero),
VMClose()
will update the file and close it. If the file could not be updated, it will return an error condition. Be warned, however, that if for some reason
VMClose()
could not finish updating a file (for example, because the disk ran out of space),
VMClose()
will return an error message, but will close the file and free the memory anyway. Thus, the most recent changes will be lost. For this reason, it is usually safer to first update the file (and handle any error messages returned) and then close it.
When GEOS shuts down, all files are closed. When it restarts, you can open the files manually.
If the file is backup-enabled, the backup blocks will be preserved until the file is next opened. That means, for example, that the next time you open the file, you can continue working on it normally, or you can immediately revert it to its condition as of the last time it was saved (in the earlier GEOS session).
VMGetMapBlock(), VMSetMapBlock()
When they're created, the blocks of a VM file are in no particular order. You will need some way to keep track of VM block handles so you can find each block when you need it. The usual way to do this is with a map block .
A map block is just like any other VM block. Like other blocks, it can be a
standard block, an LMem heap, etc. It is different in only one way: the VM manager keeps track of its handle. By calling the routine
VMGetMapBlock()
, you can get the VM handle of the map block. You can then look inside the map block to get information about the other blocks.
Note that the structure of the VM map block is entirely the concern of the creating geode. The VM manager neither requires nor specifies any internal structure or information content.
To create a map block, allocate a VM block through any of the normal techniques, then pass its VM handle as an argument to
VMSetMapBlock()
. That block will be the new map block. If there already was a map block, the old block will become an ordinary VM block.
In addition to setting a map block, you can set a
map database item
with the command
DBSetMap()
. For details, see the Database chapter.
VMGrabExclusive(), VMReleaseExclusive()
Sometimes several different geodes will need access to the same VM file. Generally, these will be several different copies of the same application, perhaps running on different machines on a network. GEOS provides three different ways shared-access can be handled.
A VM file can be one of three different types: standard, "public," and "shared-multiple." By default, all new VM files are standard. The file's type is one of its extended attributes, and can be changed with the routine
FileSetHandleExtendedAttributes()
(see the File System chapter). The document control automatically lets the user select what kind of file to create, and changes its type accordingly. (See the Document Objects chapter.)
Only one geode may write to a standard GEOS VM file at a time. If a geode has the file open for read/write access, no other geode will be allowed to open that file. If a geode has the file open for read-only access, other geodes are allowed to open it for read-only access, but not for read-write access. If a file is opened for read-only access, blocks cannot be dirtied or updated. If a geode tries to open a file for writing when the file is already open, or if the geode tries to open it for reading when the file has already been opened for writing,
VMOpen()
will return an error.
In general, when a file is opened, it is by default opened for read-write access. For example, the document control objects present a dialog box which lets the file be opened for read-only access, but has this option initially turned off. However, some files are used mainly for reference and are infrequently changed. For example, a company might keep a client address book on a network drive. Most of the time, people would just read this file; the file would only occasionally be changed. For this reason, GEOS lets you declare VM file as "public." Public files are, by default, opened for read-only access. In all other respects the file is the same as a standard GEOS VM file; it can be opened by several readers at a time, but by only one geode at a time if the geode will be writing.
Sometimes several geodes will need to be able to write to a file at once. For example, a company might have a large customer database, and several users might be writing records to the database at the same time. For this reason, GEOS lets you create "shared-multiple" files. Several geodes can have a "shared-multiple" file open at once. However, a geode cannot access the file whenever it wants. Instead, it must get the file's semaphore to access the file's data. When it needs to access the file, it calls
VMGrabExclusive()
. This routine takes four arguments:
VMGrabExclusive()
will give up trying to get the semaphore after a specified number of seconds has passed. If a timeout value of zero is passed,
VMGrabExclusive()
will block until it can get the file's semaphore.
VMOperation
enumerated type
VMOperation
values are described below.
VMGrabExclusive()
fails and times out, the operation currently being performed will be written here.
The routine returns a member of the
VMStartExclusiveReturnValue
enumerated type. The following return values are possible:
VMGrabExclusive()
failed and timed out without getting access to the file.
When a geode calls
VMGrabExclusive()
, it must pass a member of the
VMOperation
enumerated type. Most of the values are used internally by the system; while a geode should never pass these values, they may be returned by
VMGrabExclusive()
if the call times out. The following values are defined in
vm.h
:
The application may also pass any value between VMO_FIRST_APP_CODE and 0xffff. The kernel treats all these values as synonymous with VMO_WRITE; however, the application may choose to associate meanings with numbers in this range (perhaps by defining an enumerated type whose starting value is VMO_FIRST_APP_CODE).
When a thread is done accessing a file, it should release its exclusive access by calling
VMReleaseExclusive()
. The routine takes one argument, namely the file handle. It does not return anything.
VMCopyVMBlock(), VMSetReloc()
If you would like to duplicate a VM block, or copy it to another file, call
VMCopyVMBlock()
. This routine is passed three arguments:
VMFileHandle
of the file containing the source block.
VMBlockHandle
of the source block.
VMFileHandle
of the destination file (which may be the same as the source file).
The routine makes a complete duplicate of the source block, copying it to the destination file. It returns the
VMBlockHandle
of the duplicate block. Note that the duplicate block will almost certainly have a different block handle than the source block. If the block contains a copy of its own handle, you should update it accordingly.
Sometimes you will need to perform special actions whenever loading a block into memory or writing it to the disk. For example, you may store data in a compressed format on the disk, and need to expand it when it's loaded into memory. For this reason, you can set a relocation routine. This routine will be called whenever the following situations occur:
Using the routine
VMSetReloc()
, you can instruct the VM manager to call your relocation routine whenever appropriate.
A VM file is just a collection of blocks. These blocks are not kept in any particular order. If the application wants to keep the blocks in some kind of order, it must set up some mechanism to do this.
One common mechanism is the VM chain. This is simply a way of arranging blocks in a sequence, in which each block contains the handle of the next block. GEOS has a standard format for VM chains and provides utility routines which work with chains of this format.
In general usage, a "chain" is a simple tree in which each block has a link to at most one other block. However, the GEOS VM chain can also contain special "tree blocks," which may have links to any number of child blocks. By using these blocks, an application can set up VM trees of unlimited complexity.
A VM chain is composed of two kinds of blocks: chain blocks (which are linked to at most one other block), and tree blocks (which may be linked to any number of other blocks). One block is the head of the chain; chain utility routines can be passed the handle of this block, and they will act on all the blocks in the chain. If a block is a "leaf" block, it should contain a null handle. An example of a VM chain with tree blocks is shown in the figure below.
Be warned that a VM chain must not contain any circuits. That is, by following links, you should not be able to go from any block back to itself; and there should not be two different routes from any one block to any other. If you create such a VM chain and pass it to a chain utility, the results are undefined. It is your responsibility to make sure no loops occur.
A VM chain block is the same as any other VM block, with one exception: The block must begin with a
VMChainLink
structure. This structure contains a single data field,
VMC_next
, which is the handle of the next block in the chain. If the block is in a chain but has no next link,
VMC_next
is a null handle. This means, for example, that LMem heaps cannot belong to a VM chain (since LMem heaps must begin with an
LMemHeader
structure).
In addition to chain blocks, a VM chain may contain a tree block. A tree block may have several links to blocks.
A tree block begins with a
VMChainTree
structure. This structure has three fields:
VMCT_meta
VMChainLink
structure. Every block in a VM chain, including a tree block, must begin with such a structure. However, to indicate that this is a tree block, the
VMC_next
field must be set to the special value VM_CHAIN_TREE.
VMCT_offset
VMChainTree
structure and the first link. If you do not put data in this block, set this field to
sizeof(VMChainTree)
.
VMCT_count
Any of the links may be a null handle. To delete the last link in the block, just decrement
VMCT_count
(and, if you wish, resize the block). To delete a link in the midst of a block, just change the link to a null handle without decrementing
VMCT_count
. To add a new link to a VM tree block, you can either add the handle after the last link and increment
VMCT_count
; or you can replace a null handle (if there are any) with the new handle, without changing
VMCT_count
.
VMChainHandle, VMFreeVMChain(), VMCompareVMChains(), VMCopyVMChain(), VMCHAIN_IS_DBITEM(), VMCHAIN_GET_VM_BLOCK(), VMCHAIN_MAKE_FROM_VM_BLOCK()
Several utilities are provided for working with VM chains. They allow you to compare, free, or copy entire VM chains with a single command.
For convenience, all of these routines can work on either a VM chain or a database item. This is useful, because sometimes you will want to use the utility on a data structure without knowing in advance how large it will be. This way, if there is a small amount of data, you can store it in a DB item; if there is a lot, you can store it in a VM chain of any length. Whichever way you store the data, you can use the same chain utilities to manipulate it.
The routines all take, as an argument, a
VMChain
which is simply a dword. If the low word is null, the
VMChain
contains the VM block handle to a VM chain. If the low word is not null, the
VMChain
contains a
DBGroupAndItem
. Note that none of the blocks in the VM chain need be locked when the routine is called; the routine will lock the blocks as necessary, and unlock them when finished. Similarly, a DB item need not be locked before being passed to one of these routines. However, the VM file containing the structure must be open.
If you want to free an entire VM chain at once, call the routine
VMFreeVMChain()
. This routine takes two arguments, namely the VM file handle and a
VMChain
. It frees every block in the VM chain, and returns nothing.
You can compare two different VM chains, whether in the same or in different files, by calling
VMCompareVMChains()
. This Boolean routine takes four arguments, namely the handles of the two files (which may be the same) and the
VMChain
s (
i.e.
, the block handles to the two chains). The geode must have both files open when it calls this routine. The routine returns
true
(
i.e.
, non-zero) if the two chains are identical (
i.e.
, the trees have the same structures and all data bytes are identical). Note that if the chains contain tree blocks which are identical except in the order of their links, the chains will not be considered identical and the routine will return
false
(
i.e.
, zero).
You can make a copy of a VM chain with the routine
VMCopyVMChain()
. This routine copies the entire chain to a specified file, which may be the same as the source file. The blocks in the duplicate chain will have the same user ID numbers as the corresponding original blocks. The routine takes three arguments: the handle of the source file, the
VMChain
of the source chain or item, and the handle of the destination file. It copies the chain or item and returns the
VMChain
of the duplicate structure. As noted, if the low word of the
VMChain
is null, the high word will be the VM handle of the head of the chain. The geode must have both files open when it calls this routine.
Several utilities are provided for working with
VMChain
types. They are implemented as preprocessor macros, so they are very fast. The macro VMCHAIN_IS_DBITEM()
is passed a
VMChain
and returns non-zero if the VMChain identifies a DB item (
i.e.
, if the low word is non-zero) or zero if the VMChain identifies a VM chain. The macro VMCHAIN_GET_VM_BLOCK
() is passed a
VMChain
containing the block handle to a VM chain. It returns the high word,
i.e.,
the chain's VM block handle.
Finally, the macro VMCHAIN_MAKE_FROM_VM_BLOCK() takes a
VMBlockHandle
value and casts it to type
VMChain
.
Sometimes a geode needs to work with an array that can get very large. Chunk arrays are very convenient, but they are limited to the size of an LMem heap, which is slightly less than 64K; furthermore, their performance begins to degrade when they get larger than 6K. Similarly, if a geode uses raw memory for an array, it is limited to the maximum size of a global memory block, again 64K.
For this reason, GEOS provides the Huge Array library. A Huge Array is stored in a VM file. All the elements are stored in VM blocks, as is the directory information. The application can specify an array element by its index, and the Huge Array routine will lock the block containing that element and return its address. Array indices are dword-sized, meaning a Huge Array can have up to 232 elements. Since elements are stored in VM blocks, each element has a maximum size of 64K; however, the size of the entire array is limited only by the amount of disk space available. The blocks in a Huge Array are linked together in a VM chain, so the VM chain utilities can be used to duplicate, free, and compare Huge Arrays.
There are a couple of disadvantages to using Huge Arrays. The main disadvantage is that it takes longer to access an element: the routine has to lock the directory block, look up the index to find out which block contains the element, lock that block, calculate the offset into that block for the element, and return its address. (However, elements are consecutive within blocks; thus, you can often make many array accesses with a single Huge Array lookup command.) There is also a certain amount of memory overhead for Huge Arrays. Thus, if you are confident that the array size will be small enough for a single block, you are generally better off with a Chunk Array (see the Local Memory chapter).
Huge arrays may have fixed-size or variable-sized elements. If elements are variable-sized, there is a slight increase in memory overhead, but no significant slowdown in data-access time.
The huge array has two different type of blocks: a single directory block, and zero or more data blocks. The blocks are linked together in a simple (i.e., non-branching) VM chain. The directory block is the head of the chain. A Huge Array is identified by the handles of the VM file containing the array and the directory block at the head of the array.
The directory block is a special kind of LMem block. It contains a header structure of type
HugeArrayDirectory
(which begins with an
LMemBlockHeader
structure), followed by an optional fixed data area, which is followed by a chunk array.
The chunk array is an array of
HugeArrayDirEntry
structures. There is one such structure for each data block; the structure contains the handle of the data block, the size of the block, and the index number of the last element in the block.
Each data block is also a special LMem block. It contains a
HugeArrayBlock
structure (which begins with an
LMemBlockHeader
structure) and is followed by a chunk array of elements. If the Huge Array has variable-sized elements, so will each data block's chunk array.
When you want to look up a specific element, the Huge Array routines lock the directory block. They then read through the directory chunk array until they find the block which contains the specified element. At this point, the routine knows both which data block contains the element and which element it is in that block's chunk array. (For example, if you look up element 1,000, the Huge Array routine might find out that block x ends with element 950 and block y ends with element 1,020. The routine thus knows that element 1,000 in the Huge Array is in the chunk array in block y , and its element number is 50 in that block's array.)
The routine then unlocks the directory block, and locks the data block containing that element. It returns a pointer to that element. It also returns the number of elements occurring before and after that element in that chunk array. The calling geode can access all the elements in that block without further calls to Huge Array routines.
When you insert or delete an element, the Huge Array routines look up the element index as described above. They then call the appropriate chunk array routine to insert or delete the element in that data block. They then go through the directory and adjust the element numbers throughout. If inserting an element in a data block would bring the block's size above some system-defined threshold, the Huge Array routine automatically divides the data block.
When the VM routines resize an element block, they automatically make the block larger than necessary. This leaves extra space for future insertions in that block, so the block won't have to be resized the next time an element is added. This improves efficiency, since you may often be adding several elements to the same block. However, this also means that most Huge Arrays have a fair amount of unused space. If you won't be adding elements to a Huge Array for a while, you should compact the Huge Array with
HugeArrayCompressBlocks
(see Huge Array Utilities
).
Ordinarily, VM Chains may not contain LMem heaps. Huge Arrays are an exception. The reason LMem blocks cannot belong to VM chains is simple. Each block in a VM chain begins with the handle of the next block in the chain (or VM_CHAIN_TREE if it is a tree block). However, each LMem heap has to begin with an
LMemBlockHeader
structure, the first word of which is the global handle of the memory block. In order for these blocks to serve as both, special actions have to be taken. When a Huge Array block is unlocked, its first word is the handle of the next block in the chain. It is thus a VM chain block and not a valid LMem heap. When the Huge Array routine needs to access the block, it locks the block and copies the block's global handle into the first word, storing the chain link in another word. This makes the block a valid LMem heap, but it (temporarily) invalidates the VM chain.
Although VM chain utilities work on Huge Arrays, you must be sure that the Huge Array is a valid VM chain when you call the utility. In practice, this means you cannot use a VM chain utility when any block in the chain is locked or while any thread might be accessing the array. If more than one thread might be using the array, you should not use the chain utilities.
HugeArrayCreate(), HugeArrayDestroy(), HugeArrayResize(), HugeArrayLock(), HugeArrayUnlock(), HugeArrayDirty(), HugeArrayAppend(), Huge ArrayInsert(), HugeArrayReplace(), HugeArrayDelete(), HugeArrayGetCount()
GEOS provides many routines for dealing with Huge Arrays. The basic routines are described in this section. Some additional routines which can help optimize your code are described in Huge Array Utilities .
Note that you should never have more than one block of a Huge Array locked at a time. Furthermore, when you call any routine in this section (except
HugeArrayUnlock()
,
HugeArrayDirty()
, and
HugeArrayGetCount()
), you should not have
any
blocks locked. The next section contains several routines which may be used while a block is locked. Also, if you use any VM chain routines on a Huge Array, you should make sure that no blocks are locked.
To create a Huge Array, call
HugeArrayCreate()
. This routine allocates a directory block and initializes it. The routine takes three arguments:
HugeArrayDirectory
structure and the chunk array of directory entries, you can pass an argument here. The size must be at least as large as
sizeof(HugeArrayDirectory)
. Alternatively, you can pass an argument of zero, indicating that there will be no extra data area, and the default header size should be used.
HugeArrayCreate()
returns the VM handle of the directory block. This is also the handle of the Huge Array itself; you will pass it as an argument to most of the other Huge Array routines.
When you are done with a Huge Array, destroy it by calling
HugeArrayDestroy()
. This routine frees all of the blocks in the Huge Array. It takes two arguments, namely the global handle of the VM file and the VM handle of the Huge Array. It does not return anything. You should make sure that none of the data blocks are locked when you call this since all of the VM chain links must be valid when this routine is called.
To access an element in the array, call
HugeArrayLock()
. This routine takes five arguments:
The routine figures out which block has the element specified (as described above). It then locks that block. It writes a pointer to that element in the location passed, and writes the size of the element in the variable pointed to by the other pointer (this is useful if the Huge Array has variable-sized elements). It returns a dword. The more significant word is the number of consecutive elements in that block, starting with the pointer returned; the less significant word is the number of elements in that block before (and including) the element specified. For example, suppose you lock element 1,000. Let's assume that this element is in block
x
; block
x
has 50 elements, and element 1,000 in the huge array is the 30th element in block
x
. The routine would write a pointer to element 1,000 in the pointer passed; it would then return the dword 0x0015001e. The upper word (0x0015) indicates that element 1,000 is the first of the last 21 consecutive elements in the block; the lower word (0x001e) indicates that the element is the last of the first 30 consecutive elements. You thus know which other elements in the Huge Array are in this block and can be examined without further calls to
HugeArrayLock()
.
When you are done examining a block in the Huge Array, you should unlock the block with
HugeArrayUnlock()
. This routine takes only one argument, namely a pointer to any element in that block. It does not return anything. Note that you don't have to pass it the same pointer as the one you were given by
HugeArrayLock()
. Thus, you can get a pointer, increment it to work your way through the block, and unlock the block with whatever address you end up with.
 However, if you change an element, you need to dirty the block yourself or the changes won't be saved to the disk. To do this, call the routine
HugeArrayDirty()
. This routine takes one argument, namely a pointer to an element in a Huge Array. It dirties the data block containing that element. Naturally, if you change several elements in a block, you only need to call this routine once.
If you want to add elements to the end of a Huge Array, call
HugeArrayAppend()
. If elements are of uniform size, you can add up to
216
elements with one call to this routine. You can also pass a pointer to a template element; it will copy the template into each new element it creates. This routine takes four arguments:
The routine returns the index of the new element. If several elements were created, it returns the index of the first of the new elements. This index is a dword.
You can also insert one or more elements in the middle of a Huge Array. To do this, call the routine
HugeArrayInsert()
. As with
HugeArrayAppend()
, you can insert many uniform-sized elements at once, and you can pass a pointer to a template to initialize elements with. The routine takes five arguments:
It returns the index of the first of the new elements. Ordinarily, this will be the index you passed it; however, if you pass an index which is out of bounds, the new elements will be put at the end of the array, and the index returned will thus be different.
To delete elements in a Huge Array, call
HugeArrayDelete()
. You can delete many elements (whether uniform-sized or variable-sized) with one call to
HugeArrayDelete()
. The routine takes four arguments:
You can resize one of the elements by calling
HugeArrayResize()
. If you make the element smaller, then data at the end will be truncated. If you make the element larger, then the new data will be zero-initialized.
You can erase or replace the data in one or more elements with a call to
HugeArrayReplace()
. This is also the only way to resize a variable-sized element. You can pass a pointer to a template to copy into the element or elements, or you can have the element(s) initialized with null bytes. The routine takes five arguments:
You can find out the number of elements in a Huge Array with a call to
HugeArrayGetCount()
. This routine takes two arguments, namely the file handle and the handle of the Huge Array. The routine returns the number of elements in the array. Since array indices begin at zero, if
HugeArrayGetCount()
returns x, the last element in the array has index
x-1.
HugeArrayNext(), HugeArrayPrev(), HugeArrayExpand(), HugeArrayContract(), HugeArrayEnum(), HugeArrayCompressBlocks(), ECCheckHugeArray()
The routines in the other section may be all you will need for using Huge Arrays. However, you can improve access time by taking advantage of the structure of a Huge Array. As noted above, you can use any of the VM Chain routines on a Huge Array, as long as none of the blocks are locked.
If you have been accessing an element in a Huge Array and you want to move
on to the next one, you can call the routine HugeArrayNext()
.
The routine takes a pointer to a pointer to the element. It changes that
pointer to point to the next element in the array, which may be in a
different block. If the routine changes blocks, it will unlock the old block
and lock the next one. It returns the number of consecutive elements starting
with the element we just advanced to. If we were at the last element in the
Huge Array, it returns zero and the pointer will point to the last element.
If you want to move to the previous element instead of the next one, call
HugeArrayPrev()
. It also takes a pointer to a pointer to an
element. It changes that pointer to a pointer to the previous element; if
this means switching blocks, it unlocks the current block and locks the
previous one. It returns the number of elements which come before the
newly-locked one in its block, counting the newly-locked element. If the
pointer was pointing to the first element in the Huge Array, it returns
zero and the pointer will point to the first element.
If you have a block of the Huge Array locked and you want to insert an element or elements at an address in that block, call the routine
HugeArrayExpand()
. It takes three arguments:
The routine inserts the elements, dirtying any appropriate blocks. It writes a pointer to the first new element into the pointer passed. Since the data block may be resized or divided to accommodate the request, this address may be different from the one you pass. The routine returns the number of consecutive elements beginning with the one whose address was written. If the new element is in a different block from the address passed, the old block will be unlocked, and the new one will be locked.
If you have a block of a Huge Array locked and you want to delete one or more elements starting with one within the block, you can call
HugeArrayContract()
. This routine takes two arguments:
The routine deletes the elements, dirtying any appropriate blocks. It then changes the pointer to point to the first element after the deleted ones. If this element is in a different block, it will unlock the old block and lock the new one. It returns the number of consecutive elements following the one whose address was written.
You may wish to perform the same operation on a number of consecutive elements of a Huge Array.
HugeArrayEnum()
is a routine which lets you do this efficiently.
HugeArrayEnum()
takes six arguments:
HugeArrayEnum()
. The callback routine can abort the enumeration by returning
true
(
i.e.
, non-zero).
HugeArrayEnum()
calls the callback routine for every element, in order, from the first element. It passes the callback a pointer to the element and the pointer passed to
HugeArrayEnum()
. The callback routine may not do anything which would invalidate any pointers to the Huge Array; for example, it may not allocate, delete, or resize any of the elements. The callback routine should restrict itself to examining elements and altering them (
without
resizing them). The callback routine can abort the enumeration by returning
true
(
i.e.
, non-zero); if it does so,
HugeArrayEnum()
will return
true
. If
HugeArrayEnum()
finishes the enumeration without aborting, it returns
false
(
i.e.
, zero).
If the count is large enough to take
HugeArrayEnum()
past the end of the array,
HugeArrayEnum()
will simply enumerate up to the last element, then stop. For example, if you pass a start index of 9,000 and a count of 2,000, but the Huge Array has only 10,000 elements,
HugeArrayEnum()
will enumerate up through the last element (with index 9,999) then stop. However, the starting index
must
be the index of an element in the Huge Array. You can also pass a count of -1, indicatingg that
HugeArrayEnum()
should enumerate through the last element of the array. Therefore, to enumerate the entire array, pass a starting element of zero and a count of -1.
As noted above, most Huge Arrays contain a fair amount of unused space. This makes it much faster to add and remove elements, since blocks don't need to be resized very often. However, if you have a Huge Array that is not frequently changed, this is an inefficient use of space. You can free this space by calling
HugeArrayCompressBlocks()
. This routine is passed two arguments: the handle of the VM file, and the
VMBlockHandle
of the Huge Array. The routine does not change any element in the Huge Array; it simply resizes the directory and data blocks to be no larger than necessary to hold their elements. The routine does not return anything.
If you want to verify (in error-checking code) that a given VM block is the directory block of a Huge Array, you can call
ECCheckHugeArray()
. This routine is passed the VM file and block handles of the block in question. If the block is the directory block of a Huge Array, the routine returns normally; otherwise it causes a fatal error. The routine should not, therefore, be used in non-EC code.


Electronics & Computer Engineering Student - Software & Robotics passionate
Rebuilding this website, TBF soon, preview on https://thesableur.web.app
I have completed highschool at a French institution in Lebanon, before joining Imperial College London in fall 2018 to pursue an MEng degree in Electronics and Information Engineering, where I enjoy learning about and bridging the gap between hardware and software, as well as explore low-level technologies and their implementations, from signal processing to compilers.
New technologies and innovations trigger my curiosity to explore new fields in hardware and software. This passion led me to continously seek experience from professionals through summer internships or spring weeks. Recently, I've founded a startup with two friends and currently dedicating most of my time for it.
In 2014, my younger brother and I founded a robotics team from our bedroom and started competing in local FIRST competitions, making it to the World Festival in 2017 and 2018. I also regurarly attend hackathons and get involved into projects such as assitive mobile applications or a little BB8 robot.
Beside a hunger for technology-related activites, I love to practice martial arts (I've been doing Shinkyokushin Karate for over 12 years now). I am also a big fan of spanish guitar, but recently got myself into fingerstyle accoustic covers. I manage to fit in a book every 2-3 weeks as well!
I'm Jaafar, a 20 y/o Lebanese student living in London. I've always been passionate about technology, starting with robotics at a young age and catching up software development in high-school. I love to invest time and resources to improve my skills everyday, and work on projects to make our world and people's lives better. Feel free to get in touch for collaborations!


--- layout: default title: "404: Page not found" permalink: 404.html ---
you are visiting something that doesn't exist. Head back home to try finding it again. ┬─┬ノ( º _ ºノ)


Part-time Application Developer
Developing the iOS mobile application ’Waste Education Application’ with the aim of assisting students to improve the consciousness of environmental protection and waste recycling in the campus.
-
Requirement Extraction Organising a user requirement meeting with the staff from Ecoversity department of the University of Adelaide, and analysing system requirements for further system design.
-
Application Design Responsible for GUI design such as layout, theme, and styles through using StoryBoard of Swift.
-
Server Establishment Responsible for database establishment by adopting ’Django.’ framework which is implemented by Python based upon MVC pattern.
-
API (Application Interface) Design Responsible for achieving the RESTful API, connecting and enabling the communication between clients and server.
-
Quality Assurance Responsible for unit test case design for validating the code quality.
-
Server Configuration Responsible for server establishment and publishment to ensure application running smoothly, including parameter configuration, feature debugging and version backup.
Relative Skills:
-
Javascript(jQuery)
-
HTML
-
CSS
-
Swift
-
Python(Django)


---
layout: post
title: Lull point
date: '2010-04-26T21:00:00.007-04:00'
author: Pat Hyatt
description: I'm at a lull. Hear me rant.
tags:
- Vacation
- Book
- Development
- C#
- Spain
- Lull
- Boring
- Review
modified_time: '2010-04-26T22:42:38.210-04:00'
redirect_from:
- /2010/04/lull-point.html
---
I've not posted in so long, my massive audience has now probably stopped following. Sorry loyal followers!
I
must admit, I feel I just don't offer enough as a developer to post compared to other avid/interesting bloggers. And as
of recently I've been in a development lull.
No interesting discoveries, no new tools I've dabbled with, nothing
really.
Recent work has changed from a News harvester to a rewrite of a very high traffic site.
While it
is exciting/new/challenging, I'm just tanking in enthusiasm.

Recent things
I can blurb about:
-Read Don't
Make Me Think: A Common Sense Approach to Web Usability, 2nd Edition By Steve Krug.
Thoughts: Very easy read and sensible
book on usability. Life changing? No, but sometimes I think you need to get smacked in the face with obviousness to
realize things. For example the chapter on web usability I found really informative.
-Going to Spain! As we
try to do every few years, we are getting out of the U.S. for a duet of weeks to see exactly what Spain is about. Very
excited about this, and the excitement is growing as the days counting continue downward.
That's about all I
can dump off the mind at the moment. I believe it is a glimpse at my lull.
Ah well.... as Johnny Cash said,
Drive on...


Podcasts about programming are all the same. Sure, hosts and topics are different, but the format is the same. And this format – it has it's limitations.
For a long time I thought this was the way to do podcasting – you gather with your friends and discuss recent news, things you like and things you don't. Sometimes you interview people to get an outside perspective. All of this is recorded and then, after little polishing, you release it.
I think it's an excellent way to introduce listeners to new techniques and to developers in their community. This format has served well to shows like Ruby5, JSJabber, Mostly Erlang and many others. I'm one of the hosts of a podcast like that. It's called RadioJS, it's about JavaScript and it's in Russian.
On the other hand, there's a ton of subjects it's hard to talk about on these podcasts: abstract concepts like design patterns, complicated systems like the Internet and so on. Unless hosts know a lot about the topic and have spoken publicly about it, it's hard to record a coherent story in one take.
Interviews on these shows are often shallow. "What are you working on?" "How did you get into programming?" "What does your library/framework do?" Those dialogs are not very technical. I feel like they often miss an opportunity to dive deep into person's work. Also, because it's usually one guest per episode, there's no perspective. How are other people solving similar problem? What do they have to say about alternative solutions? These questions spin in my head as I listen to an episode.
There's more to podcasting
We live in a golden age of podcasting. People make so many kinds of shows: long form and short form, comedy and detective, narrative and conversational.
Several months ago I've stumbled upon a show called StartUp. It's a podcast about building a business. When I first heard it, I was amazed. The quality of the show, the story, everything was done very professionally. I`ve never heard anything like this.
Later I've discovered Ted Radio Hour, Freekonomics and Planet Money. Their format was different too. They would take one topic every episode and explore it in depth from various perspectives. I knew it was time for a different kind of a programming podcast, these shows gave me an idea about what to try first.
Significant bits
Together with my good friend Michael Beschastnov we've launched this experiment. We'd choose one coding concept every episode, take several interviews, heavily edit them, add music and try to make a story out of it. We would talk usage, implementations and alternative solutions. We would give a comprehensive overview of a technical concept in 30 minutes or less.
We believe that ideas are more important than programming languages or libraries or other tools. Once you understand the concept, it doesn't matter if it's implemented in Python or in Node or in Haskell. You get how it works. With this knowledge you can write new and debug existing code. Often you don't even have to memorize the syntax of a target language to do that.
This is our goal – share concepts in programming that we find fascinating and make an entertaining story out of it.
Explore
We love how this experiment is going so far. There're things we like about Code and things we don't, but the process of making it is super fun. Seeing listeners feedback is very rewarding.
At this moment we have released 3 episodes. They make up one series entitled "Concurrency". The show is taking shape, but we still have a backlog of ideas to try.
We have dived into one format, but there are so many others. Programming podcasts are all the same nowadays, but it doesn't have to be like that. I would encourage you to experiment. Think of an idea for a show and then just record it! Record one episode and share it with your friends and colleagues or put it out on iTunes or SoundCloud. Send it to us, we are always searching for new ways to do technical podcasting. If you don't have time to record it – send the idea anyway!
There's a lot to explore in the world of programming podcasts, let's kick off the scouting!
Andrey Salomatin
Comments !


$ npm i -g wp2vite
$ cd your_workspace/your_project
$ wp2vite
$ npm install
$ npm run vite-start
<!-- old project menu -->
build
config
docs
node_modules
public
src
components
pages
App.tsx
main.js
setupProxy.js
.eslintignore
.eslintrc.js
.gitignore
jest.config.js
package.json
package-lock.json
README.md
tsconfig.json
<!-- new project menu -->
build
config
docs
node_modules
public
src
components
pages
App.tsx
main.js
setupProxy.js
.eslintignore
.eslintrc.js
.gitignore
index.html
jest.config.js
package.json
package-lock.json
README.md
tsconfig.json
vite.config.js
A front-end project automatic conversion tool, You can make your webpack project support vite.
wp2vite will not delete your webpack configuration, but will inject vite configuration into your project to support vite.
The development speed is about 80% faster than that of webpack, and the construction speed is about 50% faster than that of webpack.


Concept TextConceptConcept for a type that can be as text of an index.
Concept for a type that can be as text of an index.
| Defined in | <seqan/sequence.h> |
|---|---|
| Signature |
concept TextConcept;
|
Detailed Description
Certain algorithms and data structures can work for both strings and string sets but need to treat these two types slightly different. Examples are index data structures and algorithms that build the indices and use the indices for lookup.
To facilitate writing of generic algorithms, the TextConcept concept gives a common interface to both for this kind of algorithms.
See Also
Interface Function Overview
-
TSeqNo getSeqNo(pos[, limits]);
Returns the sequence number of a position. -
TOffset getSeqOffset(pos[, limits]);
Returns the local sequence offset of a position. -
TPos posGlobalize(pos, limits)
Converts a local/global to a global position. -
void posLocalize(result, pos, limits)
Converts a local/global to a local position. -
void posLocalToX(dst, localPos, limits);
Converts a local to a local/global position. -
TStringSetLimits stringSetLimits(text);
Return string delimiter positions for TextConcept types.
Interface Metafunction Overview
-
SAValue<TText>::Type
The default alphabet type of a suffix array, i.e. the type to store a position of a string or string set. -
StringSetLimits<TText>::Type;
Return type of string set limits for TextConcept types.
Interface Functions Detail
TSeqNo getSeqNo(pos[, limits]);
Returns the sequence number of a position.
Parameters
pos
|
A position. Types: Pair |
|---|---|
limits
|
The limits string returned by stringSetLimits. |
Returns
TSeqNo |
A single integer value that identifies the string within the stringset pos points at. If limits is omitted or Nothing getSeqNo returns 0.If pos is a local position (of class Pair) then i1 is returned.If pos is a global position (integer type and limits is a String) then pos is converted to a local position and i1 is returned. |
|---|
Data Races
Thread safety unknown!
TOffset getSeqOffset(pos[, limits]);
Returns the local sequence offset of a position.
Parameters
pos
|
A position. Types: Pair |
|---|---|
limits
|
The limits string returned by stringSetLimits. |
Returns
TOffset |
A single integer value that identifies the position within the string pos points at.If limits is omitted or Nothing getSeqNo returns pos. If pos is a local position (of class Pair) then i2 is returned.If pos is a global position (integer type and limits is a String) then pos is converted to a local position and i2 is returned. |
|---|
Data Races
Thread safety unknown!
TPos posGlobalize(pos, limits)
Converts a local/global to a global position.
Parameters
pos
|
A local or global position (pair or integer value). Types: Pair |
|---|---|
limits
|
The limits string returned by stringSetLimits. |
Returns
TPos |
The corresponding global position of pos. If pos is an integral type pos is returned. If not, limits[getSeqNo(pos, limits)] + getSeqOffset(pos, limits) is returned. |
|---|
Data Races
Thread safety unknown!
void posLocalize(result, pos, limits)
Converts a local/global to a local position.
Parameters
pos
|
A local or global position (pair or integer value). |
|---|---|
limits
|
The limits string returned by stringSetLimits. |
result
|
Reference to the resulting corresponding local position of pos. |
Data Races
Thread safety unknown!
void posLocalToX(dst, localPos, limits);
Converts a local to a local/global position.
Parameters
dst
|
The local or global position (pair or integer value) is written here. |
|---|---|
localPos
|
The local position. |
limits
|
The string limits as returned by stringSetLimits. |
Data Races
Thread safety unknown!
TStringSetLimits stringSetLimits(text);
Return string delimiter positions for TextConcept types.
Parameters
text
|
The text to query for its string set limits. |
|---|
Returns
TStringSetLimits |
The string set limits (of type StringSetLimits). |
|---|
Data Races
Thread safety unknown!


Security
ArgentSea encourages better security practices in two ways:
- By making it easier to store password information securely.
- By enforcing the use of stored procedures / functions.
Secure Password Storage
ArgentSea’s data connection metadata divorces passwords from a legacy “connection string” implementation. Because the new .NET configuration architecture enables configuration data to be distributed among multiple sources, passwords can be stored in secure locations — independently of the rest of the connection metadata.
For example, passwords can stored in:
- UserSecrets — in a development context.
- AWS Secrets Manager
- Azure Key Vault
- Secure, encrypted file shares
Each provider has a NuGet package which simplifies loading their secrets into your application.
If your password store handles JSON, you can add credentials like this:
"Credentials": [
{
"SecurityKey": "SecKey1",
"UserName": "webuser",
"Password": "D~|Y98N?A9qkEz^u%Dhn"
},
{
"SecurityKey": "SecKey2",
"UserName": "adminUser",
"Password": "A*dNppRZq3-vE@Ci"
}
]
If your password store uses key-value pairs, you can store the same information like this:
| Key | Value |
|---|---|
| Credentials:0:SecurityKey | SecKey1 |
| Credentials:0:UserName | SecKey1 |
| Credentials:0:Password | SecKey1 |
| Credentials:1:SecurityKey | SecKey2 |
| Credentials:1:UserName | adminUser |
| Credentials:1:Password | A*dNppRZq3-vE@Ci |
Connections that use Windows authentication do not need to be stored securely. These entries can be added directly to appsettings.json:
"Credentials": [
{
"SecurityKey": "SecKey1",
"WindowsAuth": true
}
],
Stored procedures/functions
ArgentSea enforces the use of stored procedures or functions. Besides offering performance advantages, this provides substantial security benefits.
Least Privileged Access
When data access is managed by stored procedures or functions exclusively, permissions can be granted exclusively to the procedures and not to the underlying tables. If an attacker were to take ownership of the connection, they would not be able to perform any database activity that was not explicitly enabled by the existing procedures or functions.
Injection Attacks
A common data hack involves crafting the requests sent to a database server to include additional commands. Stored procedures/functions with parameters are immune to this attack, assuming no dynamic is used SQL within the procedure body.
DBA Validation
When dynamic SQL commands are created by the application, it can be difficult to debug the command sequence, even harder to correct it, and nearly impossible to constrain it to appropriate activity. When using stored procedures or functions, however, DBAs can review the implementation of dangerous commands, and ensure that their explicit use is appropriate.


Get Coverage
Get a Quote
Caluclate Your Needs
Resources
Insurance 101
Blog
Videos
Agents
About
Our Story
FAQ
Careers
Contact-Us
My Account
Coming Soon
Thankyou
Inner: 1280 x 588
Outer: 1280 x 729
Window size:
1280
x
729
Viewport size:
1280
x
588


Hey leviathans! We're Moaning Bharaath and Chatti Raghav! We're here to serve the community!
Fun
We had a lot of FUN! Just last week, the both of us almost fufiled our customer's needs! How cool? Our customer told us that our wrench was too soft and she wanted a harder one.
Other than that, she was happy with our services.
Jobs
Just look at our customer testimonials
By Jane Hoe: I am very HAPPI with rag-bath service. They made me taste the secrets of life through their service.They game me lots of pleasure
and delight, I was really thrilled! Their service opened me to a new WORLD OF FLAVOURS! GGo hire them, you'll see the light!
What are you waiting for@#%#!??
Hurry! We have not paid our taxes and the government is looking for us! Get your service before its too late!


| Property | Value |
|---|---|
| Gender | {{videoAnalytics.gender}} |
| Age | {{videoAnalytics.age_value}} ± {{videoAnalytics.age_delta}} years |
| Mood | {{videoAnalytics.mood}} |
| Features | Glasses: {{videoAnalytics.glasses}}, Beard: {{videoAnalytics.beard}}, Mustache: {{videoAnalytics.mustache}} |
| Presence | {{videoAnalytics.presence}}s with {{videoAnalytics.attention}}% attention |
| Property | Value |
|---|---|
| Device ID | {{radioAnalytics.deviceId}} |
| Associations | |
| RSSI | {{radioAnalytics.rssi}} |
| Presence | {{radioAnalytics.presence}}s |


kpsetool(1) teTeX kpsetool(1) NAME kpsetool - script to make teTeX-style kpsetool, kpsexpand, and kpsepath available SYNOPSIS kpsetool [ OPTIONS... ] COMMAND kpsexpand [ OPTIONS... ] STRING kpsepath [ OPTIONS... ] PATHTYPE DESCRIPTION kpsetool is a Bourne shell script that makes a teTeX-style kpsetool, kpsexpand, and kpsepath available. kpsetool uses web2c´s (new) kpsexpand binary, so the syntax for kpsexpand is incompatible with teTeX´s as of version 0.4. OPTIONS -m mode set Metafont mode -n progname pretend to be progname to kpathsea -p act like kpsepath(1) -v act like kpsexpand(1) -w act like kpsewhich(1) USAGE PATHTYPE can be base Metafont memory dump bib BibTeX bibliography source bst BibTeX style files cnf Kpathsea runtime configuration files dvips_config dvips config files dvips_header dvips header files fmt TeX memory dump gf generic font bitmap mem MetaPost memory dump mf Metafont source mfpool Metafont program strings mp MetaPost source mppool MetaPost program strings mpsupport MetaPost support files pict Other kinds of figures pk packed bitmap font tex TeX source texpool TeX program strings tfm TeX font metrics troff_font troff fonts vf virtual font SEE ALSO kpsewhich(1) Kpathsea: A library for path searching (info or DVI file) Web page: <http://tug.org/teTeX/> BUGS None known, but report any bugs found to<EMAIL_ADDRESS>(mailing list). AUTHOR kpsetool was written by Thomas Esser<EMAIL_ADDRESS>in March, 1997. kpsetool is in the public domain. This manual page was written by C.M. Connelly<EMAIL_ADDRESS>for the Debian GNU/Linux system. It may be used by other distributions without contacting the author. Any mistakes or omissions in the manual page are my fault; inquiries about or corrections to this manual page should be directed to me (and not to the primary author). teTeX September 2000 kpsetool(1)


Super should be called at the end of the method. Example :
public class DummyActivity extends Activity { public void onPause() { foo(); // missing call to super.onPause() } }


Currently, the compiler can generate code for either the C language or the Java language. Using the C code is described in the "cmanual.html" file.
The file will contain a single class named either "X" or named as defined by the "java_class" option. All messages in the file X.proto will be translated to static public classes inside the top-level class. If those messages themselves define nested classes, the they will be defined inside the corresponding translated message class. This means that the natural naming mechanism "X.y.z" is supported.
Enumeration declarations are also inserted in the appropriate class in the output file to correspond with the location of the enumeration in the .proto file.
In addition, and for every message declaration, message M {...} for example, the file will contain methods to serialize an instance of a message M (M.write()), to deserialize an instance of a message M (M m = M.read()), and to compute the run-time size of an instance of a message M (M.getSize()).
message Msg { required float field1 = 1; optional uint64 field2 = 2; repeated int32 field3 = 3; }This will generate the following output.
public class Main { static public class Msg { float field1; Long field2; Integer[] field3; };It will also generate the following three methods
The getSize method is necessary because the actual size of each Msg object may vary at run-time.
Protobuf enumeration declarations are also translated in the more or less obvious way. So given the protobuf enumeration
enum Testenum { ECON1 = 1; ECON2 = 2; ECON3 = 3; ECON4 = 4; ECON5 = 5; ECON6 = 6; }it will be translated to the following Java enumeration declaration.
public enum Testenum { ECON1(1), ECON2(2), ECON3(3), ECON4(4), ECON5(5), ECON6(6); private int value; static public int getValue() {return value;} public Testenum(int value) {this.value = value;} public Testenum toEnum(int i) { switch (i) { case 1: return Testenum.ECON1; case 2: return Testenum.ECON2; case 3: return Testenum.ECON3; case 4: return Testenum.ECON4; case 5: return Testenum.ECON5; case 6: return Testenum.ECON6; default: return null; } } Testenum;The toEnum code is provided to allow the generated code to properly convert an on-the-wire integer representation of an enum value to the corresponding enum constant.
javac -classpath ".;/usr/local/lib/ast.jar" Test.java X.javaThen execute using some variant of this command.
The stream to be used is determined by the direction, which in turn is determined by the mode of enum type unidata.ast.runtime.AbstractIO.IOmode.
AbstractIO defines a set of methods.
Note that users will rarely call these operators; rather, they are used by internally by ASTRuntime.
The runtime system provides one implementation of AbstractIO, namely ByteIO. This class reads and writes to byte arrays (byte[]). If no outputstream is specified, then the default is an instance of java.io.ByteArrayOutputStream.
The class constructor for the generated message classes expect an instance of the runtime object as the argument.
A close() method is provided to reclaim resources.
A number of methods are provided to support, primarily, the reading and writing of primitive typed fields. As a rule, the user will not call these methods, but understanding them is useful when trying to understand the generated code. These method prototypes are as follows.
static int test(Msg* msg) { int status = AST_NOERR; ASTRuntime* rt; bytes_t encoded; Msg* decodedmsg = NULL; /* Create an expandable byteio stream into which msg will be encoded. */ /* Note the use of NULL buffer and 0 length */ /* This function creates an ASTRuntime instance and returns a pointer in the last argument. */ status = ast_byteio_new(AST_WRITE,NULL,0,&rt); if(status != AST_NOERR) goto fail; /* Write the msg object into the runtime buffer */ status = Msg_write(rt,msg); if(status != AST_NOERR) goto fail; /* Extract the contents of the stream */ status = ast_byteio_content(rt, &encoded); if(status != AST_NOERR) goto fail; /* Reclaim the ASTRuntime object */ status = ast_reclaim(rt); if(status != AST_NOERR) goto fail; /* Create byteio stream backed by the bytes from the encode process. */ status = ast_byteio_new(AST_READ,encoded.bytes,encoded.nbytes,&rt); if(status != AST_NOERR) goto fail; /* Decode a Msg instance; not the use of the Msg** in last argument */ decodedmsg = NULL; status = Msg_read(rt,&decodedmsg); if(status != AST_NOERR) goto fail; status = ast_reclaim(rt); if(status != AST_NOERR) goto fail; /* Compare using an assumed user-defined compare function*/ if(!compare(msg,decodedmsg)) { fprintf(stderr,"*** FAIL: content mismatch\n"); status = AST_EFAIL; goto fail; } fprintf(stderr,"*** PASS: %s\n",testname); return status; fail: fprintf(stderr,"*** FAIL: %s status=%d\n",testname,status); return status; }


Belgium hand surprise call-up to Cobbaut
BRUSSELS: Anderlecht left back Elias Cobbaut received a surprise call-up to the Belgium squad to play Russia and Cyprus this month as coach Roberto Martinez on Friday named a 29-man selection for their last two Euro 2020 qualifiers.
The 21-year-old defender makes the squad for the first time in place of injured veteran Jan Vertonghen, who with 118 appearances is Belgium's most capped international.
Cobbaut was picked above Premier League-based defenders Bjorn Engels and Christian Kabasele, and Zinho Vanheusden of Standard Liege, who had all been in contention to replace both Vertonghen and Thomas Meunier, who has a hamstring tear.
Winger Maxime Lestienne, who was called up last month for the first time in six years but then had to withdraw because of injury, gets another comeback chance while forward Leandro Trossard is also back in the squad but Adnan Januzaj misses out.
Belgium, top of the FIFA world rankings, have won all eight of their Group I games and last month became the first team to secure qualification to the Euro 2020 finals.
They could still be toppled from first place in the group by Russia, who host them in St Petersburg next Saturday. Russia are second with seven wins from their eight matches.
Belgium complete their campaign at home to Cyprus in Brussels on Nov. 19.
Squad:
Goalkeepers: Thibaut Courtois (Real Madrid), Simon Mignolet (Club Brugge), Matz Sels (Racing Strasbourg), Hendrik van Crombrugge (Anderlecht)
Defenders: Toby Alderweireld (Tottenham Hotspur), Dedryck Boyata (Hertha Berlin), Elias Cobbaut (Anderlecht), Jason Denayer (Olympique Lyonnais), Leander Dendoncker (Wolverhampton Wanderers), Brandon Mechele (Club Brugge), Thomas Vermaelen (Vissel Kobe)
Midfielders: Yannick Carrasco (Dalian Yifang), Timothy Castagne (Atalanta), Nacer Chadli (Anderlecht), Kevin de Bruyne (Manchester City), Thorgan Hazard (Borussia Dortmund), Dennis Praet, Youri Tielemans (both Leicester City), Hans Vanaken (Club Brugge), Axel Witsel (Borussia Dortmund)
Forwards: Michy Batshuayi (Chelsea), Christian Benteke (Crystal Palace), Eden Hazard (Real Madrid), Maxime Lestienne (Standard Liege), Romelu Lukaku (Inter Milan), Dries Mertens (Napoli), Divock Origi (Liverpool), Leandro Trossard (Brighton and Hove Albion), Yari Verschaeren (Anderlecht).
(Reporting by Mark Gleeson; Editing by Christian Radnedge)


Things that confused me about cross-entropy
26 Dec 2020Every once in a while, I try to better understand cross-entropy by skimming over some Medium posts and StackExchange answers. I always come away only half-understanding it.
A major cause of confusion is that different sources use different notations and conventions. Here are some that tripped me up.
p and q vs y and p
In information theory, the cross-entropy for an event with \(M\) discrete outcome classes is
\[H(p, q) = -\sum_{j=1}^M{p_j log (q_j)}\]… where \(p\) is the true distribution of outcomes and \(q\) is the approximating distribution.
But in machine learning, the cross-entropy is defined as:
\[H(y, p) = -\sum_{j=1}^M{y_j log (p(y_j))}\]… where \(y\) are the true labels and \(p(y)\) is the approximating distribution, i.e. the classifier’s probability predictions.
Notice that the meaning of \(p\) has been reversed! In information theory, it is the true distribution. In machine learning, it is the approximating distribution.
This points to a key difference between generic information theory and machine learning applications. In information theory, the target distribution \(p\) is a probability distribution where the probability mass can be distributed broadly over the classes. In machine learning, the target distribution \(y\) has all of its mass allocated to a known label.
Multiple classes vs multiple events, and indicators vs labels
As shown above, the cross-entropy for a single event with \(M\) classes is:
\[H(y, p) = -\sum_{j=1}^M{y_j log (p(y_j))}\]If there are multiple events, you just need to sum up the cross-entropies from all the events so that the total cross-entropy is:
\[H(y, p) = -\sum_{i=1}^N \sum_{j=1}^M{y_j^{(i)} log (p(y_j^{(i)}))}\]… where \(N\) is the number of events.
This all makes notational sense. Unfortunately, confusion arises when people use non-compact notation for summing over binary classes. When dealing with binary classification, it’s common to see cross-entropy defined as:
\[H(y, p) = -\sum_{i=1}^N{y_i log (p(y_i)) + (1-y_i) log (1 -p(y_i))}\]Superficially, this looks a lot like the first formula, but it’s actually just a clever and somewhat confusing way of writing the second formula for binary classes. Its notation differs from previous formulas in a few ways:
- First, this formula sums over events, whereas the first formula in this section only sums over classes.
- Second, because it is summing over events, it uses \(p(y_i)\) rather than \(p(y_j)\). Here, \(p(y_i)\) refers to the probability of a positive result on an event. In previous formulas, \(p(y_j)\) referred to the probability of the particular class being considered in the summation loop for that event.
- Third, it’s easier to think of \(y_i\) (when not wrapped by a \(p\)) as a label, rather than an indicator variable. When the event has an outcome of 1, \(y_i\) is 1. When it has an outcome of 0, \(y_i\) is 0. In previous formulas, as you loop through classes you would set \(y_j\) to be 1 whenever the outcome was the class being considered; otherwise you set it to 0.
The image below summarizes the many confusing differences between these formulas.
Where to learn about cross-entropy
I don’t know if there is a single best place to learn about cross-entropy, but below are a few places that were helpful. If you read them make sure you think about (a) how they are notating the true distribution and the approximating distribution (b) whether they are writing about single events or multiple events (c) whether they are writing about binary classes or multi classes and (d) whether the \(y\)’s are indicators or labels.
- Statistical Rethinking. Information theory notation, single event case, multiple classes.
- Andrew Webb. Both notations, both the single and multiple event case, multiple classes.
- Harshith. ML notation, multiple events, binary classes.


Each year, the CDC publishes a dataset of deaths from that year. Approximately 2.5 million people die in the US each year and each one gets a row in the dataset with a slew of demographic data. Below , I have grouped all deaths from 2003, 2008, and 2013 by their highest educational attainment. The vertical axis represents the average age of death for each group. From this data, a question arises: Why do 8th grade graduates live longer than their more educated fellows?
Why do those who only reach an elementary level of education live longer than their more educated peers? Are they dying of something different? To answer that question, the below visualization shows the cause of death by educational attainment. If those whose highest educational attainment was elementary school died of something different, the circles, representing the percentage of an educational group that died of a particular cause, should change in size from row to row. Mouse over the circles for exact percentages, and use the controls if you want to drill down further into the data.


{% extends 'layouts/base.html' %} {% set page_title = _('Server error') %} {% block body %}
Oopsie! Looks like something went wrong. Please try and restart the software, if the problem persists, please email support at<EMAIL_ADDRESS>You can provide logs from the following locations:
Please also provide the steps to reproduce your issue
{% include 'errors/help_block.html' %}


SFU student starts fundraiser to hire charter bus as transit shutdown looms
Grayson Lee created the fundraiser after his post in a SFU carpooling group on Facebook received traction
by Katya SlepianA Simon Fraser University student has started a fundraiser to hire a charter bus to take people to the Burnaby campus ahead of a full shutdown of Metro Vancouver’s bus and SeaBus system Wednesday.
The shutdown is set to run Wednesday to Friday this week and comes as negotiations between the Coast Mountain Bus Company and the union representing its workers remain broken down.
Grayson Lee created the fundraiser after his post in a SFU carpooling group on Facebook received traction.
“I’m thinking of renting a 50 seater bus to go from production up and down the mountain on Wed – Fri between school hours, if there’s enough interest. It’d cost between 3-4$ per trip (much cheaper than the $15 cab ride). If you are interested please comment below,” Lee wrote on Nov. 21.
The post had 28 comments and 79 likes as of Monday morning, and a GoFundMe page was set up Sunday.
The fundraiser aims to raise $3,000 for the charter bus, which Lee said the company would require upfront.
“I do not have the funds to make that payment by myself. I’m hoping for SFU alumni’s, students, and anyone who this impacts, if they could pitch in whether $2, $5, any donation counts to cover the expenses of the bus, as otherwise we not able to make this a reality,” Lee wrote.
The bus would run from Wednesday to Friday at peak school hours, which Lee defines as 7:30 a.m. to 10 a.m. and 4 p.m. to 6 p.m.
As of Monday morning, $150 has been raised.
READ MORE: $150M sticking point: Coast Mountain, Unifor fight over wage gap as transit lockout looms


News
Articles
Archives
Calendar
This Week
This Month
Workshops
Lessons
Special Events
Lounge
Discussion
Private Messages
Search
Scene
Local Venues
Local Bands
Instruction
Groups
Lessons
Workshops


Welcome to Recogniser!
Write a digit in the black square. The system will attempt to guess the number and display it here: {{ guess }}