Interface SyncApi
-
@Path("/_matrix/client/r0") @Consumes("application/json") @Produces("application/json") public interface SyncApi
To read events, the intended flow of operation is for clients to first call the /sync API without a since parameter. This returns the most recent message events for each room, as well as the state of the room at the start of the returned timeline. The response also includes a next_batch field, which should be used as the value of the since parameter in the next call to /sync. Finally, the response includes, for each room, a prev_batch field, which can be passed as a start parameter to the /rooms/<room_id>/messages API to retrieve earlier messages.
-
-
Nested Class Summary
Nested Classes Modifier and Type Interface Description static class
SyncApi.Presence
PresenceContent.
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description void
events(String from, Long timeout, String roomId, javax.ws.rs.core.UriInfo uriInfo, javax.ws.rs.core.HttpHeaders httpHeaders, javax.ws.rs.container.AsyncResponse asyncResponse, javax.ws.rs.core.SecurityContext securityContext)
This will listen for new events related to a particular room and return them to the caller.void
sync(String filter, String since, Boolean fullState, String setPresence, Long timeout, javax.ws.rs.core.UriInfo uriInfo, javax.ws.rs.core.HttpHeaders httpHeaders, javax.ws.rs.container.AsyncResponse asyncResponse, javax.ws.rs.core.SecurityContext securityContext)
Synchronise the client's state with the latest state on the server.
-
-
-
Method Detail
-
sync
@GET @Path("/sync") void sync(@QueryParam("filter") String filter, @QueryParam("since") String since, @QueryParam("full_state") Boolean fullState, @QueryParam("set_presence") String setPresence, @QueryParam("timeout") Long timeout, @Context javax.ws.rs.core.UriInfo uriInfo, @Context javax.ws.rs.core.HttpHeaders httpHeaders, @Suspended javax.ws.rs.container.AsyncResponse asyncResponse, @Context javax.ws.rs.core.SecurityContext securityContext)
Synchronise the client's state with the latest state on the server. Clients use this API when they first log in to get an initial snapshot of the state on the server, and then continue to call this API to get incremental deltas to the state, and to receive new messages.
Requires auth: Yes.
Return:SyncResponse
.Status code 200: The initial snapshot or delta for the client to use to update their state.
- Parameters:
filter
- The ID of a filter created using the filter API or a filter JSON object encoded as a string. The server will detect whether it is an ID or a JSON object by whether the first character is a "{" open brace. Passing the JSON inline is best suited to one off requests. Creating a filter using the filter API is recommended for clients that reuse the same filter multiple times, for example in long poll requests.since
- A point in time to continue a sync from.fullState
- Controls whether to include the full state for all rooms the user is a member of.
If this is set to true, then all state events will be returned, even if since is non-empty. The timeline will still be limited by the since parameter. In this case, the timeout parameter will be ignored and the query will return immediately, possibly with an empty timeline.
If false, and since is non-empty, only state which has changed since the point indicated by since will be returned.
By default, this is false.setPresence
- Controls whether the client is automatically marked as online by polling this API. If this parameter is omitted then the client is automatically marked as online when it uses this API. Otherwise if the parameter is set to "offline" then the client is not marked as being online when it uses this API. One of: ["offline"]timeout
- The maximum time to wait, in milliseconds, before returning this request. If no events (or other data) become available before this time elapses, the server will return a response with empty fields.
By default, this is 0, so the server will return immediately even if the response is empty.uriInfo
- Request Information.httpHeaders
- Http headers.asyncResponse
- Asynchronous response.securityContext
- Security context.
-
events
@GET @Path("/events") void events(@QueryParam("from") String from, @QueryParam("timeout") Long timeout, @QueryParam("room_id") String roomId, @Context javax.ws.rs.core.UriInfo uriInfo, @Context javax.ws.rs.core.HttpHeaders httpHeaders, @Suspended javax.ws.rs.container.AsyncResponse asyncResponse, @Context javax.ws.rs.core.SecurityContext securityContext)
This will listen for new events related to a particular room and return them to the caller. This will block until an event is received, or until the timeout is reached.
This API is the same as the normal /events endpoint, but can be called by users who have not joined the room.
Note that the normal /events endpoint has been deprecated. This API will also be deprecated at some point, but its replacement is not yet known.
Requires auth: Yes.
Return:Page
of theEvent
s.Status code 200: The events received, which may be none.
Status code 400: Bad pagination from parameter.
- Parameters:
from
- The token to stream from. This token is either from a previous request to this API or from the initial sync API.timeout
- The maximum time in milliseconds to wait for an event.roomId
- The room ID for which events should be returned.uriInfo
- Request Information.httpHeaders
- Http headers.asyncResponse
- Asynchronous response.securityContext
- Security context.
-
-


@ProviderType
public interface ThreadPoolMBean
| Modifier and Type | Method and Description |
|---|---|
String |
getBlockPolicy()
Retrieve the block policy of the thread pool.
|
int |
getExecutorActiveCount()
Retrieve the active count from the pool's Executor.
|
long |
getExecutorCompletedTaskCount()
Retrieve the completed task count from the pool's Executor.
|
int |
getExecutorCorePoolSize()
Retrieve the core pool size from the pool's Executor.
|
int |
getExecutorLargestPoolSize()
Retrieve the largest pool size from the pool's Executor.
|
int |
getExecutorMaximumPoolSize()
Retrieve the maximum pool size from the pool's Executor.
|
int |
getExecutorPoolSize()
Retrieve the pool size from the pool's Executor.
|
long |
getExecutorTaskCount()
Retrieve the task count from the pool's Executor.
|
long |
getKeepAliveTime()
Return the configured keep alive time.
|
int |
getMaxPoolSize()
Return the configured maximum pool size.
|
long |
getMaxThreadAge()
Return the configured max thread age.
|
int |
getMinPoolSize()
Return the minimum pool size.
|
String |
getName()
Return the name of the thread pool
|
String |
getPid()
Return the configuration pid of the thread pool.
|
String |
getPriority()
Return the configured priority of the thread pool.
|
int |
getQueueSize()
Return the configured queue size.
|
int |
getShutdownWaitTimeMs()
Return the configured shutdown wait time in milliseconds.
|
boolean |
isDaemon()
Return whether or not the thread pool creates daemon threads.
|
boolean |
isShutdownGraceful()
Return whether or not the thread pool is configured to shutdown gracefully.
|
boolean |
isUsed()
Return whether or not the thread pool is in use.
|
String getBlockPolicy()
int getExecutorActiveCount()
long getExecutorCompletedTaskCount()
int getExecutorCorePoolSize()
int getExecutorLargestPoolSize()
int getExecutorMaximumPoolSize()
int getExecutorPoolSize()
long getExecutorTaskCount()
long getMaxThreadAge()
long getKeepAliveTime()
int getMaxPoolSize()
int getMinPoolSize()
String getName()
String getPid()
String getPriority()
int getQueueSize()
int getShutdownWaitTimeMs()
boolean isDaemon()
boolean isShutdownGraceful()
boolean isUsed()
Copyright © 2017 The Apache Software Foundation. All rights reserved.


Cubs Guessing Game!
Click on the four icons of our beloved Cubs, which all have a secret value, in order to match the Target Number.


Distributed Training with Uneven Inputs Using the Join Context Manager¶
Author: Andrew Gu
참고
View and edit this tutorial in github.
참고
Join
is introduced in PyTorch 1.10 as a prototype feature. This
API is subject to change.
In this tutorial, you will see:
An overview of the Join context manager.
An example of how to use the context manager with
DistributedDataParallel
.An example of how to use the context manager with both
DistributedDataParallel
andZeroRedundancyOptimizer
.An example of passing in keyword arguments to the context manager.
A dive into how the Join context manager works.
An example showing how to make a toy class compatible with the context manager.
Requirements¶
What is Join
?¶
In Getting Started with Distributed Data Parallel - Basic Use Case, you saw the general skeleton for using DistributedDataParallel to perform data parallel training. This implicitly schedules all-reduces in each backward pass to synchronize gradients across ranks. Such collective communications require participation from all ranks in the process group, so if a rank has fewer inputs, then the other ranks will hang or error (depending on the backend). More generally, this problem persists for any class that performs per-iteration synchronous collective communications.
Join
is a context manager to be used around your per-rank training loop to
facilitate training with uneven inputs. The context manager allows the ranks
that exhaust their inputs early (i.e. join early) to shadow the collective
communications performed by those that have not yet joined. The ways in which
the communications are shadowed are specified by hooks.
Using Join
with DistributedDataParallel
¶
PyTorch’s DistributedDataParallel works out-of-the-box with the Join
context manager. Here is an example usage:
import os
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.distributed.algorithms.join import Join
from torch.nn.parallel import DistributedDataParallel as DDP
BACKEND = "nccl"
WORLD_SIZE = 2
NUM_INPUTS = 5
def worker(rank):
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '29500'
dist.init_process_group(BACKEND, rank=rank, world_size=WORLD_SIZE)
model = DDP(torch.nn.Linear(1, 1).to(rank), device_ids=[rank])
# Rank 1 gets one more input than rank 0
inputs = [torch.tensor([1]).float() for _ in range(NUM_INPUTS + rank)]
num_inputs = 0
with Join([model]):
for input in inputs:
num_inputs += 1
loss = model(input).sum()
loss.backward()
print(f"Rank {rank} has exhausted all {num_inputs} of its inputs!")
def main():
mp.spawn(worker, nprocs=WORLD_SIZE, join=True)
if __name__ == "__main__":
main()
This produces the following output (where the print()
s from rank 0 and
rank 1 may be arbitrarily ordered):
Rank 0 has exhausted all 5 of its inputs!
Rank 1 has exhausted all 6 of its inputs!
참고
DistributedDataParallel provided its own join() context manager
prior to the introduction of this generic Join
context manager. In the
above example, using with Join([model]):
is equivalent to using
with model.join():
. One limitation of the existing
DistributedDataParallel.join()
is that it does not allow multiple
participating classes, e.g. DistributedDataParallel
and
ZeroRedundancyOptimizer together.
Using Join
with DistributedDataParallel
and ZeroRedundancyOptimizer
¶
The Join
context manager works not only with a single class but also with
multiple classes together. PyTorch’s ZeroRedundancyOptimizer
is also
compatible with the context manager, so here, we examine how to modify the
previous example to use both DistributedDataParallel
and
ZeroRedundancyOptimizer
:
from torch.distributed.optim import ZeroRedundancyOptimizer as ZeRO
from torch.optim import Adam
def worker(rank):
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '29500'
dist.init_process_group(BACKEND, rank=rank, world_size=WORLD_SIZE)
model = DDP(torch.nn.Linear(1, 1).to(rank), device_ids=[rank])
optim = ZeRO(model.parameters(), Adam, lr=0.01)
# Rank 1 gets one more input than rank 0
inputs = [torch.tensor([1]).float() for _ in range(NUM_INPUTS + rank)]
num_inputs = 0
# Pass both `model` and `optim` into `Join()`
with Join([model, optim]):
for input in inputs:
num_inputs += 1
loss = model(input).sum()
loss.backward()
optim.step()
print(f"Rank {rank} has exhausted all {num_inputs} of its inputs!")
This will yield the same output as before. The notable change was
additionally passing in the ZeroRedundancyOptimizer
instance into
Join()
.
Passing Keyword Arguments¶
Classes may provide keyword arguments that modify their behavior in the context
manager at run time. For example, DistributedDataParallel
provides an
argument divide_by_initial_world_size
, which determines if gradients are
divided by the initial world size or by the effective world size (i.e. number
of non-joined ranks). Such keyword arguments can be passed directly into the
context manager.
with Join([model, optim], divide_by_initial_world_size=False):
for input in inputs:
...
경고
The keyword arguments passed into the context manager are shared across
all participating classes. This should not be a limitation since we do
not expect cases where multiple Joinable
s need differing settings
of the same argument. Nonetheless, this is something to keep in mind.
How Does Join
Work?¶
Now that we have seen some preliminary examples of how to use the Join
context manager, let us delve deeper into how it works. This will provide a
greater insight into the full capability that it offers and prepare you to make
your own custom classes compatible. Here, we will go over the Join
class as
well as the supporting classes Joinable
and JoinHook
.
Joinable
¶
To begin, classes compatible with the Join
context manager must inherit
from the abstract base class Joinable
. In particular, a Joinable
must
implement:
join_hook(self, **kwargs) -> JoinHook
This returns the JoinHook
instance for the Joinable
, determining how
joined processes should shadow the per-iteration collective communications
performed by the Joinable
.
join_device(self) -> torch.device
This returns a device to be used by the Join
context manager to perform
collective communications, e.g. torch.device("cuda:0")
or
torch.device("cpu")
.
join_process_group(self) -> ProcessGroup
This returns the process group to be used by the Join
context manager to
perform collective communications.
In particular, the join_device
and join_process_group
are required
attributes to ensure that the context manager can schedule collective
communications between joined and non-joined processes. One usage is to count
the number of non-joined processes on each iteration using an all-reduce.
Another usage is for implementing the mechanism required for
throw_on_early_termination=True
, which we will explain later below.
DistributedDataParallel
and ZeroRedundancyOptimizer
already inherit
from Joinable
and implement the above methods, which is why we could
directly use them in the previous examples.
Joinable
classes should make sure to call the Joinable
constructor
since it initializes a JoinConfig
instance, which is used internally by
the context manager to ensure correctness. This will be saved in each
Joinable
as a field _join_config
.
JoinHook
¶
Next, let us break down the JoinHook
class. A JoinHook
provides two
entry points into a context manager:
main_hook(self) -> None
This hook is called repeatedly by each joined rank while there exists a rank
that has not yet joined. It is meant to shadow the collective communications
performed by the Joinable
in each training iteration (e.g. in one forward
pass, backward pass, and optimizer step).
post_hook(self, is_last_joiner: bool) -> None
This hook is called once all ranks have joined. It is passed an additional
bool
argument is_last_joiner
, which indicates if the rank was one of
the last to join. The argument may be useful for synchronization.
To give concrete examples of what these hooks may look like, the provided
ZeroRedundancyOptimizer
main hook performs an optimizer step per normal
since the joined rank is still responsible for updating and synchronizing its
shard of the parameters, and the provided DistributedDataParallel
post-hook
broadcasts the final updated model from one of the last joining ranks to ensure
that it is the same across all ranks.
Join
¶
Finally, let us examine how these fit into the Join
class itself.
__init__(self, joinables: List[Joinable], enable: bool = True, throw_on_early_termination: bool = False)
As we saw in the previous examples, the constructor takes in a list of the
Joinable
s that participate in the training loop. These should be the
classes that perform collective communications in each iteration.
enable
is a bool
that can be set to False
if you know that there
will not be uneven inputs, in which case the context manager becomes vacuous
similar to contextlib.nullcontext()
. This also may disable join-related
computation in the participating Joinable
s.
throw_on_early_termination
is a bool
that can be set to True
to
have each rank raise an exception the moment that uneven inputs are detected.
This is useful for cases that do not conform to the context manager’s
requirements, which is most typically when there are collective communications
from different classes that may be arbitrarily interleaved, such as when using
DistributedDataParallel
with a model that has SyncBatchNorm
layers. In
such cases, this argument should be set to True
so that the application
logic can catch the exception and determine how to proceed.
The core logic occurs in the
__exit__()
method, which loops while there exists a non-joined rank, calling eachJoinable
〈s main hook, and then once all ranks have joined, calls their post hooks. Both the main hooks and post-hooks are iterated over in the order that theJoinable
s are passed in.The context manager requires a heartbeat from non-joined processes. As such, each
Joinable
class should make a call toJoin.notify_join_context()
before its per-iteration collective communications. The context manager will ensure that only the firstJoinable
passed in actually sends the heartbeat.
경고
As mentioned above regarding throw_on_early_termination
, the
Join
context manager is not compatible with certain compositions of
classes. The Joinable
〈s JoinHook
s must be serializable since each
hook is fully executed before proceeding to the next. In other words, two
hooks cannot overlap. Moreover, currently, both the main hooks and post-
hooks are iterated over in the same deterministic order. If this appears to
be a major limitation, we may modify the API to permit a customizable
ordering.
Making a Toy Class Work with Join
¶
Since the previous section introduced several concepts, let us see them in
practice with a toy example. Here, we will implement a class that counts the
number of inputs that are seen across all ranks before its rank joins. This
should provide a basic idea of how you may make your own class compatible
with the Join
context manager.
Specifically, the following code has each rank print out (1) the number of inputs across all ranks that seen before it joins and (2) the total number of inputs across all ranks.
import os
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.distributed.algorithms.join import Join, Joinable, JoinHook
BACKEND = "nccl"
WORLD_SIZE = 2
NUM_INPUTS = 5
class CounterJoinHook(JoinHook):
r"""
Join hook for :class:`Counter`.
Arguments:
counter (Counter): the :class:`Counter` object using this hook.
sync_max_count (bool): whether to sync the max count once all ranks
join.
"""
def __init__(
self,
counter,
sync_max_count
):
self.counter = counter
self.sync_max_count = sync_max_count
def main_hook(self):
r"""
Shadows the counter's all-reduce by all-reducing a dim-1 zero tensor.
"""
t = torch.zeros(1, device=self.counter.device)
dist.all_reduce(t)
def post_hook(self, is_last_joiner: bool):
r"""
Synchronizes the max count across all :class:`Counter` s if
``sync_max_count=True``.
"""
if not self.sync_max_count:
return
rank = dist.get_rank(self.counter.process_group)
common_rank = self.counter.find_common_rank(rank, is_last_joiner)
if rank == common_rank:
self.counter.max_count = self.counter.count.detach().clone()
dist.broadcast(self.counter.max_count, src=common_rank)
class Counter(Joinable):
r"""
Example :class:`Joinable` that counts the number of training iterations
that it participates in.
"""
def __init__(self, device, process_group):
super(Counter, self).__init__()
self.device = device
self.process_group = process_group
self.count = torch.tensor([0], device=device).float()
self.max_count = torch.tensor([0], device=device).float()
def __call__(self):
r"""
Counts the number of inputs processed on this iteration by all ranks
by all-reducing a dim-1 one tensor; increments its own internal count.
"""
Join.notify_join_context(self)
t = torch.ones(1, device=self.device).float()
dist.all_reduce(t)
self.count += t
def join_hook(self, **kwargs) -> JoinHook:
r"""
Return a join hook that shadows the all-reduce in :meth:`__call__`.
This join hook supports the following keyword arguments:
sync_max_count (bool, optional): whether to synchronize the maximum
count across all ranks once all ranks join; default is ``False``.
"""
sync_max_count = kwargs.get("sync_max_count", False)
return CounterJoinHook(self, sync_max_count)
@property
def join_device(self) -> torch.device:
return self.device
@property
def join_process_group(self):
return self.process_group
def find_common_rank(self, rank, to_consider):
r"""
Returns the max rank of the ones to consider over the process group.
"""
common_rank = torch.tensor([rank if to_consider else -1], device=self.device)
dist.all_reduce(common_rank, op=dist.ReduceOp.MAX, group=self.process_group)
common_rank = common_rank.item()
return common_rank
def worker(rank):
assert torch.cuda.device_count() >= WORLD_SIZE
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '29500'
dist.init_process_group(BACKEND, rank=rank, world_size=WORLD_SIZE)
counter = Counter(torch.device(f"cuda:{rank}"), dist.group.WORLD)
inputs = [torch.tensor([1]).float() for _ in range(NUM_INPUTS + rank)]
with Join([counter], sync_max_count=True):
for _ in inputs:
counter()
print(f"{int(counter.count.item())} inputs processed before rank {rank} joined!")
print(f"{int(counter.max_count.item())} inputs processed across all ranks!")
def main():
mp.spawn(worker, nprocs=WORLD_SIZE, join=True)
if __name__ == "__main__":
main()
Since rank 0 sees 5 inputs and rank 1 sees 6, this yields the output:
10 inputs processed before rank 0 joined!
11 inputs processed across all ranks!
11 inputs processed before rank 1 joined!
11 inputs processed across all ranks!
Some key points to highlight:
A
Counter
instance performs a single all-reduce per iteration, so the main hook performs a single all-reduce as well to shadow it.The
Counter
class makes a call toJoin.notify_join_context()
at the beginning of its__call__()
method since that is a place before its per- iteration collective communications (i.e. its all-reduce).The
is_last_joiner
argument is used to determine the broadcast source in the post-hooks.We pass in the
sync_max_count
keyword argument to the context manager, which is then forwarded toCounter
〈s join hook.


June
Hunger
South London Forever
Big God
Sky Full Of Songs
Grace
Patricia
100 Years
The End Of Love
No Choir


Go to Payment Page
Your NEM Address:
XEM for verification:
Your Name (Follow NRIC):
Amount of tokens to purchase:
5
10
20
Checkbox is CHECKED!
Namespace id:
Name of token to purchase (Verify before pressing add)
Add to cart
Submit and proceed to payment
Shopping Cart


WITNESS LIST
Finance
April 7, 2015 9:00 AM
FOR:
Bayless, Jr., James L. Attorney (Texas State Cemetery Committee), Austin, TX
Registering, but not testifying:
FOR:
Reynolds, Joseph H. Retired (Bull Creek Road Coalition)
Whatley, Tom Retired (Bull Creek Road Coalition)
ON:
Walker, Jason Director of Research (Texas State Cemetery)
Registering, but not testifying:
FOR:
Garza, Steven Governmental Affairs Representative (Texas Association of Realtors)
Gonzalez, Daniel Consultant (Texas Association of Realtors), Austin
Scurlock, Stephen Associate Executive (Independent Bankers Association of Texas),
Austin
Young, Chris lawyer (Linebarger Goggan Blair & Sampson, LLP), Austin, TX
ON:
Wood, Tanya Eriksson Lawyer (Linebarger Goggan Blair & Sampson, LLP), Houston, TX
FOR:
Andro, Richard Senior Property Tax Consultant (Refund Advisory Corp), Dallas, TX
Registering, but not testifying:
FOR:
Allen, George B. Executive Vice President (Texas Apartment Association), Austin
Andro, Elena Rohweder Communications (Self), Allen, TX
Garza, Steven Governmental Affairs Representative (Texas Association of Realtors)
Morrison, Chet Property Tax Consultant (Texas Assoc. of Property Tax Professionals),
Austin, TX
Ramirez, Melissa Lawyer (Popp Hutcheson), Austin
FOR:
Popp, Jim Attorney (Texas Association of Realtors/self), Austin
South, Brent Chief Apprasier Hunt CAD (Texas Association of Appraisal Districts),
Greenville, TX
AGAINST:
Amezquita, Michael Chief Appraiser Bexar CAD (Myself & Bexar Appraisal Dist.), SA,
TX
ON:
Crigler, Marya Chief Appraiser (also providing written testimony) (Travis Central
Appraisal District), Austin, TX
Lankford, Alvin Chief Appraiser (Williamson Central Appraisal District)
WITNESS LIST
Finance
April 7, 2015 9:00 AM
Registering, but not testifying:
FOR:
Allen, George B. Executive Vice President (Texas Apartment Association), Austin
Cartwright, Deborah Lawyer (Harris County Appraisal District), Houston, TX
Christian, George attorney (Texas Taxpayers and Research Association), Austin, TX
Garza, Steven Governmental Affairs Representative (Texas Association of Realtors)
Giarratani, Dominic Assistant Director (Texas Association of School Boards), Austin
Gonzalez, Daniel Consultant (Texas Association of Realtors), Austin
Igo, Shanna Deputy Executive Director (Texas Municipal League)
Joslove, Scott President & CEO (Texas Hotel & Lodging Association), Austin, TX
Laborde, Blanca (The Real Estate Council/NAIOP), Austin
LeBas, James tax analyst (Tex. Apartment Ass'n, Chemical Council, TxOGA, Tex. Ass'n of
Manufacturers), Austin, TX
Lehman, Mark Vice President - Texas Association of Realtors (Texas Association of
Realtors), Austin
Minick, Steven Vice President , Gov't Affairs (Texas Association of Business), Austin, TX
Morrison, Chet (Texas Association of Property Tax Proff.), Austin, TX
Norman, Scott Executive Director (Texas Association of Builders), Austin, TX
Partida, Neftali Government Affairs Consultant (Texas BOMA), Austin, TX
Ramirez, Melissa Lawyer (Popp Hutcheson), Austin
Reed, Charles Assistant Administrator (Dallas County Commissioners Court)
Simpson, Stephanie Vice President (Tx Association of Manufacturers), ATX
Warndof, Donna Director of Legislative Relations (Harris County), Houston, TX
AGAINST:
Lavine, Dick Sr Fiscal Analyst (Center for Public Policy Priorities)
ON:
Artearle Supervisor - Revenue Accounting - Electronic Reporting (Texas Comptroller of
Public Accounts)
Cantu, Anthony Area Manager - Revenue Administration (Texas Comptroller of Public
Accounts)
FOR:
Frazier, Kyle Lobbyist (Texas Wine & Grape Growers), Austin, TX
Hyman, Mark President/CEO Llano Estacado Winery (also providing written testimony)
(Myself/Our Winery), Colleyville, TX
Registering, but not testifying:
FOR:
Heath, Brian Business (winery) Owner (Self), Fredericksburg, TX
Stevens, Bill Capitol Dome Partners (Self), Austin
Yates, Ron Winery/Vineyard (Self), Spicewood, TX
ON:
Hunter, Dan Asst. Commissioner (Texas Department of Agriculture), Austin
WITNESS LIST
Finance
April 7, 2015 9:00 AM
Lineberger, R. Daniel Professor & Head of Horticultural Sciences (Texas A&M Agrilife
Extension Service), College Station, TX
Reichek, Karen Director, Contracts & Grants - Tx Dept of Agriculture (Tx Dept of
Agriculture)
Rosenbaum, Coy revenue estimator - subject matter expert (Texas Comptroller of Public
Accounts)
FOR:
Jackson, Richie CEO (Texas Restaurant Association)
Registering, but not testifying:
FOR:
Goldman, Adam Lobbyist (Darden Restaurants)
Greenberg, Steve VP of Governmental Affairs (Landry's), Houston, TX
Hamilton, Mike Consulting (Brinker International), Austin
Joslove, Scott President & CEO (Texas Hotel and Lodging Association), Austin, TX
ON:
Dillard, Don Tax Policy Expert (Texas Comptroller of Public Accounts)
FOR:
Sheer, Jim Texas Retailers Association, VP Govt Relations (Texas Retailers, Association),
Austin, TX
Registering, but not testifying:
FOR:
Garza, Steven Governmental Affairs Representative (Texas Association of Realtors)
Gonzalez, Daniel Consultant (Texas Association of Realtors), Austin
Howe, Billy State Legislative Director (Texas Farm Bureau), Austin, TX
Lieberknecht, Chloe Director of Government Relations (The Nature Conservancy), Austin,
TX
Metzger, Luke Director (Environment Texas), Austin
Norman, Scott Executive Director (Texas Association of Builders), Austin, TX
Reed, Cyrus Conservation Director (Sierra Club, Lone Star Chapter), Austin
Weinberg, David Texas League of Conservation Voters (Texas League of Conservation
Voters), Austin, TX
Yarbrough, Brian Attorney (The Home Depot), Austin, TX
Registering, but not testifying:
ON:
Cantu, Anthony Area Manager - Electronic Reporting (Texas Comptroller of Public
Accounts)
Registering, but not testifying:
FOR:
Igo, Shanna Deputy Executive Director (Texas Municipal League)
WITNESS LIST
Finance
April 7, 2015 9:00 AM
ON:
Brewster, Whitney Executive Director (Texas Department of Motor Vehicles), Austin, TX
Earle, Art Supervisor - Electronic Reporting (Texas Comptroller of Public Accounts)
Flores, Linda CFO (Texas Department of Motor Vehicles), Austin, TX
Registering, but not testifying:
ON:
Montemayor, Piper Public Finance Manager (Comptroller of Public Accounts)
FOR:
LeBas, James tax analyst (Tx OGA), Austin, TX
Registering, but not testifying:
FOR:
Christian, George attorney (Texas Taxpayers and Research Association), Austin, TX
Esparza, Servando Manager, Government Affairs (Texas Independent Producers and
Royalty Owners' Association)
Greytok, John Attorney (National Association of Mass Appraisal (NAMA)), Austin
Hinkle, Dan Lobbyist (BP), Sugar Land, TX
Shepperd, Ben (Permian Basin Petroleum Association)
South, Brent Chief Appraiser Hunt CAD (Texas Association of Appraisal Districts),
Greenville, TX
Stevens, Bill govt affairs consultant (Texas Alliance of Energy Producers)


Open For
Smoke Options
Features
All
All
Breakfast
Lunch
Dinner
Late Night
Cigars
Cigars
Vaporizers
Cigarettes
Alcohol Served
Alcohol Served
Cash Only
Free Wi-Fi
Indoor Smoking
Lounge
Outdoor Smoking
Tobacco Sales
Sort by distance
Sort by distance
Sort by rating


About Me
Hey there! My name is Manjinder and I'm a Full Stack Web Developer.
I am currently a Software Engineering student at the University of Strathclyde in Glasgow, where I am learning more about computer science, especially the logic, the functions and everything that stands behind any programming language.
I started programming since I was in high school where I learnt basic web programming languages such as HTML5 and CSS3. Nowadays I keep updating myself by learning new languages and deliver new projects that could help me to improve my coding experience by making my code more efficient and reliable. In my free time, I always try to learn new trends in order to keep the pace in this fast-changing environment.


Decided to do some 5-HTP recovery a few months ago, four days after taking 65 mg 2C-D. Big mistake.
Surprise #1: Wake up 2 hours after taking 5-HTP, to find that I have beautiful closed-eye visuals. Apparently, 2C-D takes a long time to get out of my system. Something I'd already gotten hints of on my first session with 50 mg. Enjoy visuals for a minute, go back to sleep. Everything OK.
At 2pm next day, when I'm awake for four hours: suddenly find out that things are `weird'. Words don't make sense. The things that they represent are `normal', it's just the words that are weird. `Yellow' is a perfectly normal color, but a strange word. Communication with others is normal; maybe a bit more sparkling. Behaviour is normal. Odd feelings go away after about 2 hours, only to resurface about 6-7 hours later, for the same period. Overall, feels very much like Shulgin described it in Pihkal's part 1, `fugue state'.
Sad insight: when doing some cleaning up, I notice that I behave like a very complicated automaton; I'm just reacting to external stimuli. Rationalization of my actions lags behind half a second. Free will is a joke.
Next day, same regime, only a bit more intense. Am glad that so much of my behavior is automated. Have small anxiety attacks centered about the thought `what happens if I have to do something which isn't automated', `what should I do if people notice?' and `would I recognize it in time if I do something incredibly stupid, or which could get myself injured?' Feels like insanity. Insanity is strange.
Funny thing is, memory works OK. I did remember my PIN correctly for a bank account that I hadn't used in five years to withdraw money from it. Remembering shopping lists (short term memory) wasn't a problem either.
After a few days, things slowly bounced back to the normal, thank goodness. Five days later, I was `the old me' again. I regard this experience with mixed feelings: in some aspects, it was absolutely terrifying. OTOH, I have more respect for people who have to live in this condition forever.
Another note: On the second day of this experience, I could suppress the `oddness' and accompanying anxiety by taking two vodka's; on the third day, the comforting effect of alcohol was substantially less.
So this is one for the pharmalogical guru's out there: It seems to me that 2C-D does something on the re-uptake side, for a loooong time. Any suggestions?
Experience Reports are the writings and opinions of the individual authors who submit them.
Some of the activities described are dangerous and/or illegal and none are recommended by Erowid.


Is located at 4375 N Trenton, or about 200 yards north of the intersection of N Alaska and N Trenton (Bus Stop ID: 5862). Walk north along the sidewalk adjacent to Village Gardens' Seeds of Harmony Community Garden to reach the orchard.


Match Summary
The Match Summary dialog displays a summary view of the search results. The number of matches and number
of files with matches is displayed at the top of the window. The list box has one line for each matching
file and displays:
Total
the total number of matches in the file
Data
the total number of matches in the file that occur in the data sections
Code
the total number of matches in the file that occur in the code sections
Comment
the total number of matches in the file that occur only in comments
A left mouse click on the column headers will cause the list to be resorted by that column. The first sort will be ascending, click again to sort in descending order.


The Prototype History Research Tool is being built using existing description of archival resources. While much of the processing is successful, a significant portion of the data is inaccurate. The resource description data was not created with the current processing objectives in mind. The quality of the resource descriptions is uneven, most often because of simple human error and oversight, either in the data itself or in the encoding of the data. There are typographical errors in names and dates, and errors in data encoding (for example, labeling an organization a person, or vice-versa). Some times the differences are based on conflicting evidence, such as differences in spelling or life dates.
While refining the computational techniques used continues, such techniques alone will always fall short. The most fundamental problem is identifying when similar names are for the same person or different persons. Even for human editors, identity resolution can be an exceptional challenge and sometimes cannot be reliably achieved due to insufficient or ambiguous evidence.
Users will frequently find two or more SNAC identity descriptions for the same person or organization. In attempting to resolve identities, SNAC has intentionally erred on the side of not combining similar identities when there is uncertainty. The rationale for this bias is that it can be exceptionally difficult for users to identify when distinct identities have been incorrectly combined. SNAC thus attempts to avoid such combinations, even at the expense of not combining what to human users are seemingly obvious descriptions for the same person or organization. When two identities are similar but do not cross a certainty threshold, SNAC relates them to one another and labels them “maybeSameAs.” Nevertheless, users will discover identity descriptions that combine two or more identities.
The associated resource descriptions typically make a distinction between when an identity created the resources, and when the identity is referenced in the resources. In some cases, the entity may be represented as the creator of linked archival resources when in fact, the person, family, or organization is not the primary creator of the collection, but merely the creator of some portion of it, perhaps even just one or two items out of hundred or thousands of items.
For many of the archival resources described in WorldCat, we were unable to determine the name of the holding institutions based on identifying codes. Thus the descriptions of the resources list the holding institution as “unknown” followed by the code. In most cases, the linked WorldCat record will identify the holding institution.
We plan to address this issue in the near future, with the objective of being able to display on a map the geographic location of holdings to assist researchers in planning travel.
While most of the persons appearing as related to others persons are socially (including family relations) or professionally related, some are intellectually related, which is to say, were not known to one another or perhaps not even contemporaries, but are related because of influence or, quite frequently, because a person collected items by and about a person who lived before him or her.
Because archivists and librarians described the creators of archival resources in the description of the resources, the identity descriptions frequently contained multiple biographical notes for each entity, derived from the multiple source resource descriptions. Again because of encoding errors, users will find with some frequency a biography for one entity associated with a different entity. Upon closer examination, there will almost always be some relation between the two entities. Another common error is due to the archivists or librarian incorrectly labeling abstracts of resources as biographical data. Thus some of the "biographies" are simply not biographies.


Technologies
See how the right platforms and the right people can help you do your best work
Series: C0D3RS
ShuDon Brown aims to bring other young women into tech
Now a senior at William Peace University studying business analytics, ShuDon was one of dozens of P-TECH students to graduate at just 16 years old with a high school diploma, an associate tech degree and a paid IBM internship on her resume.
The program sparked her passion for coding, and showed her the power of technology and its ability to connect people and change lives. Now poised on the brink of an exciting new-collar career, ShuDon is just beginning to explore her potential.


{% extends 'base.html'%} {% import 'bootstrap/wtf.html' as wtf%} {% block content %} {{super()}}
{{error}}
Don't have an account? Sign up here


Large scale enterprise services like SQL Server and Exchange Server can be memory hungry beasts. Given the chance they will devour as much RAM as you can feed them, using it for caching to reduce costly I/O requests. This kind of service is often deployed on some Big Iron hardware with the sole aim of allowing it free rein of the host machine - its job being to serve clients, preferably as many as possible, and in the shortest possible time.
This article will outline the various memory constraints that affect 32-bit processes on the Windows platform and the solutions that both Intel and Microsoft provide for overcoming them through hardware, OS configuration or API changes.
When Windows NT was first being developed back in the early 1990’s you were lucky to find hard disks with a capacity over 2GB, let alone that much physical RAM. The initial design decision was to split the 4GB virtual address space that every 32-bit process would be limited to into two halves. That meant 2GB was reserved for the system (or kernel space) and 2GB for the application (or user space). Even today this address space limit of 4GB is still in effect for 32-bit processes. What the Windows engineers have done instead is provide a variety of techniques to either shuffle the kernel/user allocation ratio about or provide other APIs to allow larger memory regions to be allocated and different portions of that to be mapped into the process address space on demand [Russinovich].
Much of the confusion around this particular topic is due to the differences between the following limits: the virtual address space that a process is bound by, the amount of physical RAM that is defined by the hardware and the disk-based virtual memory provided by additional page-files. In some older articles the terms “memory”, “virtual memory” and “address space” are used interchangeably which only compounds the confusion. So, to ensure consistency throughout this article I am going to provide clear definitions of these key constraints.
Process Virtual Address Space
A process lives within a 4GB virtual address space. The limit mirrors that of a 32-bit pointer and is deemed “virtual” because the address pointer does not refer to physical memory but is actually a logical address. Instead, the page belonging to the address can be mapped anywhere within physical RAM or even inside a page-file. This is the classic “Level of Indirection” at play.
Physical Memory
Naturally this is the hardware you have within your machine. Desktop editions of Windows have historically only allowed you to access up to 4GB, whereas the Data Centre Edition of Windows Server supports up to 64GB.
Page-Files
The hard disk can also act as a temporary store for memory pages that are not currently in use. This is called virtual memory because it can only be used for page storage – the pages still have to be present in physical memory to be accessed.
This total size of all paging files defines the virtual memory limit for the entire machine; this can be smaller or larger than the per-process 4GB limit.
Commit Charge (Total Memory)
If you open the Task Manager and look at the “Performance” tab you will see a number of system memory figures quoted. One of them is labelled “Commit Charge”. This represents the sum of both the Physical RAM and any space allocated via page-files. It is the total amount of memory available for all processes.
Reserved & Committed Process Pages
Within a process the pages that constitute the virtual address space can be in one of three states – Free, Reserved or Committed. Free pages are exactly that – pages which have yet to be used. Committed pages are those that are in use and count towards the process’s footprint as they must be backed either by the page-file (for data) or the executable image (for code). The intermediate state of Reserved is a half-way house used to put a region of address space to one side without actually forcing the OS to commit any physical resources to maintaining it (except for bookkeeping).
Reserved memory is a particularly tricky beast because it is invisible in the Task Manager due to there being no physical overhead and yet it creates contention and fragmentation that is difficult to observe without inspecting the process directly.
There are essentially two ways that you can run out of memory. The first is to exhaust your own process’s virtual address space by utilising all the pages within it, or making it impossible for the heap manager to find enough free contiguous pages from which to satisfy a memory allocation request. The second method involves consuming all available system memory (i.e. both physical and virtual) so that the OS cannot allocate a free memory page to any process. The implication of the latter is that a different process is the cause of a memory allocation failure – you might only be the victim.
To diagnose a process breaching its own
limits you can monitor it with PERFMON.EXE
and watch the Process |
Virtual Bytes and Process | Private Bytes counters. The former represents the
amount of virtual address space that has ever been allocated for heaps,
page-file sections, executable code, etc. The latter is the number of Committed
Pages which represents the footprint of the process within the total memory
available to the system.
The Private Bytes counter can also be seen in Task Manager under the confusingly named column “VM Size”. Alternatively Process Explorer, via the Properties | Performance tab, provides a single dialog for a process that contains all the important memory statistics. However it uses the term “Virtual Size” in place of “Virtual Bytes”. The following table maps the terms between the various common tools:-
|
Tool |
Working Set |
Commit Charge |
Address Space |
|
Task Manager |
Mem Usage |
VM Size |
N/A |
|
Perfmon |
Working Set |
Private Bytes |
Virtual Bytes |
|
Process Explorer |
Working Set |
Private Bytes |
Virtual Size |
As we shall see later when discussing the mechanisms for breaking the 4GB barrier the column name “Commit Charge” becomes less meaningful; but it is a good first-order approximation.
The exact cause of the process’s exhaustion will likely need much closer examination of the actual page usage, for which WinDbg can be of great assistance. Another more recent tool from the Sysinternals stables, called VMMap, can also be of use. The latter is more graphical in nature than WinDbg so is easier for visualisation.
Determining that the entire machine has hit
the buffers can be a much simpler affair. Bring up the Performance tab in Task
Manager and compare the
There are a number of different options available for remedying a memory bound 32-bit process that range from simple OS level configuration changes to architectural changes via the use of certain Win32 APIs. Porting to 64-bit Windows is mentioned here as well, but only out of completeness.
We start with the OS/process configuration based solutions as they don’t require any code changes per-se.
One of the simplest ways to increase the
amount of memory a process can use on 32-bit Windows is to enable the /3GB flag
in the Window’s booot.ini
file. This has the effect of adjusting the kernel/user address
space split in favour of the application by 1GB, i.e. instead of a 2GB/2GB split
you have a 3GB/1GB split. The downside to this is that the kernel address space
is halved so there is less space for certain key kernel data structures such as
the number of System Page Table Entries mentioned earlier. The /USERVA flag is
an alternative to /3GB that allows for fine tuning of this ratio.
Unfortunately this magic flag is no good by
itself. The increase in application address space means that all of a sudden an
application could start dealing with addresses above 0x7FFFFFFF. Signed pointer
arithmetic on memory allocated above this threshold could expose latent bugs
that may lead to subtle data loss instead of catastrophic failure. Consequently
an application or service must declare itself compatible with this larger
address space by being marked with the /LARGEADDRESSAWARE flag in the
executable image. This flag is accessible via the Visual C++ linker and is
exposed by the later editions of the Visual Studio IDE under System | Linker |
Enable Large Addresses. For .Net applications you currently need to use a
custom build step that invokes EDITBIN.EXE
to set the flag.
To aid in testing your application’s
compatibility with high addresses there is a flag (MEM_TOP_DOWN
)
that can be passed to VirtualAlloc()
to force higher addresses to be allocated before lower ones (the
default).
As an aside the more recent documentation from Microsoft on this topic now uses the term 4-Gigabyte Tuning (4GT) [MSDN].
Extending the virtual address space of a single process overcomes one limitation, but there is a second one in play on 32-bit Windows that affects your ability to run many of these “large address aware” processes, such as in a Grid Computing environment. The maximum amount of physical memory that could be managed by Windows was also originally 4GB. This is still the case for the 32-bit desktop editions of Windows, but the server variants are able to address much more physical RAM – up to 64 GB on the Data Centre Server edition.
This has been achieved by utilising an Intel technology known as Physical Address Extensions which was introduced with the Pentium Pro. It adds an extra layer to the page table mechanism and extends entries from 32-bits to 64-bits so that up to 128GB could theoretically be addressed.
The introduction of PAE means that kernel
drivers would now also be exposed to physical addresses above the 4GB barrier,
something they may not have originally been tested for. Windows tries to keep
buffers under the 4GB limit to aid reliability, but once again the enablement
of the feature must be a conscious one – this time via the /PAE switch also in boot.ini
. If
the server hardware supports Hot Add Memory this flag is actually enabled by
default.
As always there is a cost to enabling this and the halving of the Page Table Index from 10-bits to 9-bits via /PAE means that there are half as many System Page Table Entries available for use. If you combine this with the /3GB flag you will have significantly reduced this resource and may see the server straining badly under heavy I/O load, i.e. you could start seeing those 1450 and 1453 errors mentioned earlier.
The other major casualty is the video adaptor [Chen], but this is often of little consequence as application servers are not generally renowned for their game playing abilities. Of course the rise in general-purpose graphics processing units (GPGPU) puts a different spin on the use of such hardware in modern servers.
Naturally all this /3GB and /PAE nonsense goes away under 64-bit Windows as the total system address space is massive by comparison. Although in theory you have 64-bits to play with, implementation limitations mean there are actually only 48-bits to work with. Still, 256 TB should be enough for anyone?
But, 64-bit Windows doesn’t just benefit 64-bit processes; the architecture also changes the address space layout for 32-bit processes too. The kernel address space now lives much higher up leaving the entire 4GB region for the application to play with (assuming that your image is marked with the /LARGEADDRESSAWARE flag as before).
The obvious solution to all these shenanigans might just simply be to recompile your application as a 64-bit process. Better still, if you rewrite it in .Net you have the ability to run as either a 32-bit or 64-bit process as appropriate with no extra work. Only, it’s never quite that simple…
There are many issues that make porting to a 64-bit architecture non-trivial, both at the source code level, and due to external dependencies. Ensuring your pointer arithmetic is sound and that any persistence code is size agnostic are two of the main areas most often written about. But you also need to watch those 3rd party libraries and COM components as a 64-bit process cannot host a 32-bit DLL, such as an inproc COM server.
The hardware and operating system will also behave differently. There are plenty of “gotchas” waiting to catch you out during deployment and operations. In the corporate world 32-bit Windows desktops are still probably the norm with 64-bit Windows becoming the norm in the server space. So, whilst the 64-bit editions of SQL & Exchange Server are well bedded-in, custom applications are still essentially developed on a different platform.
Having a user address space of 2, 3, or even 4 GB does not of course mean that you get to use every last ounce. You’ll have thread stacks and executable image sections taking chunks out. Plus, if you use C++ and COM you have at least two heaps competing, both of which will hold to ransom any virtual address descriptors (VADs) that they reserve, irrespective of whether they are in use or not. Throw in “Virtual Address Space Fragmentation” and you’re pretty much guaranteed (unless you’ve specifically tuned your application’s memory requirements) to get less than you bargained for.
The following table describes my experiences of the differences between the maximum and realistic usable memory for a process making general use of both the COM and CRT heaps:-
|
Max User
Address Space |
Useable Space |
|
2.0 GB |
1.7 GB |
|
3.0 GB |
2.6 GB |
|
4.0 GB |
3.7 GB |
This kind of information is useful if you want to tune the size of any caches, or if you need to do process recycling such as in a grid or web-hosted scenario. To see the amount of virtual address space used by a process you can watch the “Virtual Bytes” Perfmon counter as described earlier.
Those who went through the 16-bit to 32-bit Windows transition will no doubt be overly cautious - the risk/reward for porting a line-of-business application that is only in need of a little more headroom may not be sufficient to justify the cost and potential upheaval straight away.
If it’s caching you need, and you don’t mind going out-of-process on the same machine (or even making a remote call) then there are any number of off-the-shelf products in the NOSQL space, such as the open source based Memcached. However if you’re looking to do something yourself and you want to avoid additional dependencies, or need performance closer to in-process caching, then there are two options – Address Windowing Extensions and Shared Memory.
What you need to bare in mind though is that it’s not possible to overcome the 4 GB address space limit, but what both these mechanisms allow is the ability to store and access more than 4GB memory very quickly – just not all at exactly the same time.
Windows 2000 saw the addition of a new API targeted specifically at this problem, and it is the one SQL Server uses. The AWE API is designed solely with performance in mind and provides the ability to allocate and map portions of the physical address space into a process. As the name implies you cannot directly access all that memory in one go but need to create “windows” onto sections of it as and when you need to. The number and size of windows you can have mapped at any one time is still effectively bound by the 4GB per-process limit.
Due to the way the AWE work there are some restrictions on the memory that is allocated:-
The API functions allow you to allocate memory as raw pages (as indicated by the use of the term Page Frame Numbers) – this is the same structure the kernel itself uses. You then request for a subset of those pages to be mapped into a region of the process’s restricted virtual address space to gain access to it, using the previously returned Page Frame Numbers.
For services such as SQL Server and Exchange Server, which are often given an entire host, this API allows them to make the most optimal use of the available resources on the proviso that the memory will never be paged out.
There is another way to access all that extra memory using the existing Windows APIs in a manner similar to the AWE mechanism, but without many of its limitations – Shared Memory. Apart from not needing any extra privileges the memory allocated can also be paged which is useful for overcoming transient spikes or exploiting the paging algorithm already provided by the OS.
Allocating shared memory under Windows is
the job of the same API used for Memory Mapped Files. In essence what you are
mapping is a portion of a file, though
not an application defined file but a part of the system’s page-file. This is
achieved by passing INVALID_HANDLE_VALUE
instead of a real file handle to CreateFileMapping()
.
The example below creates a shared segment of 1MB:-
const size_t
size = 1024 * 1024;
HANDLE
mapping = CreateFileMapping(INVALID_HANDLE_VALUE, NULL,
PAGE_READWRITE, 0u, size, NULL);
if (mapping
== NULL)
throw std::runtime_error("Failed to
create segment");
At this point we have allocated a chunk of
memory from the system, but we can’t access it. More importantly though we
haven’t consumed any of our address space either. To read and write to it we
need to map a portion (or all) of it into our address space, which we do with MapViewOfFile()
. When we’re done we can free up the address space again with UnmapViewOfFile()
. Continuing our example we require the following code to access the
shared segment:-
const size_t
offset = 0;
const size_t
length = 1024 * 1024;
void* region
= MapViewOfFile(mapping, FILE_MAP_ALL_ACCESS, 0u, 0u,
length);
if (region
== NULL)
throw std::runtime_error("Failed to
map segment");
// read
& write to the region...
UnmapViewOfFile(region);
Every time we need to access the segment we
just map a view, access it and un-map the view again. When we’re completely
done with it we can free up the system’s memory with the usual call to CloseHandle()
.
This approach is not without its own
constraints as anyone who has used VirtualAlloc()
will
know. Just as with any normal heap allocation the actual size will be rounded
up to some extent to match the underlying page size. What is more restrictive
though is that the “window” you map to access the segment (via MapViewOfFile
) must start on an offset which is a multiple of the “allocation
granularity”. This is commonly 64K and can be obtained by calling GetSystemInfo()
. The length can be any size and will be rounded up to the nearest
page boundary. The pretty much guarantees it’s only useful with larger chunks
of data.
A more subtle problem can arise if you fail
to match the calls to MapViewOfFile
with those to UnmapViewOfFile
. Each call to MapViewOfFile
bumps the reference count on the underlying segment handle and so
calling CloseHandle
will not free the segment if any views are still mapped. If left
unchecked this could create one almighty memory leak that would be interesting
to track down.
Apart from the API limitations there is also the problem of not being able to cache or store raw pointers to the data either outside or inside the memory block – you must use or store offsets instead. The base address of each view is only valid for as long as the view is mapped so care needs to be taken to avoid dangling pointers.
One other operational side-effect of this technique that you need to warn your System Administrators about is the massive rise in page faults that they will see in the process stats. What they need to understand is that these are probably just “soft faults” where a physical page is mapped into a process and not a “hard fault” where a disk access also occurs. Although the segment is officially backed by the system page-file if enough physical RAM exists the page should never be written out to disk and so provides excellent performance.
I have previously used shared memory segments very successfully in two 32-bit COM heavy services that ran alongside other services on a 64-bit Windows 2003 server. One of them cached up to 16 GB of data without any undue side effects, even when transient loads pushed it over the physical RAM limit and some paging occurred for short periods.
I’m currently working on a .Net based system that is dependent on a 32-bit native library and have earmarked the technique again as one method of overcoming out-of-memory problems caused by needing to temporarily cache large intermediate blobs of data.
The ability to cache data in shared memory, which is effectively reference counted by the OS, provided the basis for a prototype mechanism that would allow multiple “engine” processes running on the same host to cache common data without needing a separate service process to act as a gateway. This would avoid the massive duplication of cached data for each process, which, as the number of CPUs (and therefore engine processes) increased, would afford more efficient use of the entire pool of system RAM.
The mechanism was fairly simple. Instead of each engine process storing its large blobs of common data in private memory, it would be stored in a shared segment (backed by a deterministic object name) and mapped on demand. The use of similarly (deterministically) named synchronisation objects ensures that only one engine needed to request the data from upstream and the existence of a locally cached blob could be detected easily too. This was done by exploiting the fact that creation of an object with the same name as another succeeds and returns the special error ERROR_ALREADY_EXISTS.
The idea was prototyped but never used in production as far as I know.
This article provided a number of techniques to illustrate how a 32-bit Windows process can access more memory that the 2GB default. These ranged from configuration tweaks involving the /3GB and /PAE flags through to the AWE and Shared Memory APIs. Along the way it helped explain some of the terminology and showed how to help diagnose memory exhaustion problems.
Thanks to Matthew Wilson for reciprocating and commenting on my first draft, and to Frances Buontempo for her valuable feedback and encouragement too.
[Chen] Raymond Chen, Kernel address space consequences of the /3GB switch,
http://blogs.msdn.com/b/oldnewthing/archive/2004/08/06/209840.aspx.
[MSDN] Memory Limits for Windows Releases, http://msdn.microsoft.com/en-gb/library/windows/desktop/aa366778(v=vs.85).aspx
[Russinovich] Mark Russinovich and David Solomon, Windows Internals 4th edition
Chris Oldwood
07/01/2013
Biography
Chris started out as a bedroom coder in the 80s, writing assembler on 8-bit micros. These days it’s C++ and C# on Windows in big plush corporate offices. He is also the commentator for the Godmanchester Gala Day Duck Race and can be contacted via gort@cix.co.uk.


-
Applications assigned
-
Applicants responded to feedback
-
Assessors accepted
- Assign applications Assign applications
- Invite assessors
You can allocate applications to assessors once they have responded.
- Allocate applications to assessors Allocate applications to assessors


DfE enquiries
Central newsdesk 020 7783 8300
General enquiries 0370 000 2288
News story
Education Secretary Michael Gove has today made a statement in relation to school funding and the Priority School Building Programme.
In tackling the challenges we face on school building I have been determined to use the capital funding at my disposal to best effect, seeking value for money and efficiency from every pound spent. Sebastian James’ review of capital recommended a complete overhaul of the system for allocating capital investment so that we can focus on the repair and refurbishment of schools in the greatest need alongside meeting the pressure for new, good school places.
Over the past two years we have allocated £2.7 billion to local authorities to support the provision of new school places and £2.8 billion for the maintenance of the school estate to meet the needs of maintained schools and academies. Over the spending review period, total capital investment will be over £17 billion.
In addition, last year I invited bids to a new programme from schools in need of urgent repair. 587 schools applied for the programme on the basis of their condition need. Today I can confirm that 261 schools will be rebuilt, or have their condition needs met through the Priority School Building programme (PSBP) and a copy of the list has been placed in the House Libraries. Officials have today written to all schools who applied for the programme to confirm whether their application has been successful. Work will begin immediately and the first schools will be open in 2014.
I recognise that many of the schools that applied to the PSBP and have been unsuccessful will also have significant condition needs. Some of those will have their needs addressed through the other funding we have made available for maintenance. Where that is not the case, I will use the information from the national programme of surveys we are currently conducting to ensure that, subject to funds available in the next spending review period, those schools which need renovation will have their needs addressed as quickly as possible. By next autumn we will have details about the condition of every school in the country. Information on the condition of all schools was last collated centrally in 2005.
I know that many schools will be disappointed not to be included in the programme. We have had to take difficult decisions in order to target spending on those schools that are in the worst condition. In order to ensure that the process was robust and fair, a qualified surveyor has visited every school for which an eligible application was received to verify the condition of the buildings. This was necessary to make sure the schools being taken forward are those with the greatest overall condition need.
The condition need of some schools is so severe that urgent action is necessary. I have decided to make a limited amount of capital grant available to address the needs of the highest priority schools in the programme. 42 schools - those in the very worst condition and all special schools included within the programme - will be taken forward straight away using capital grant. It is right that the condition needs of special schools - where some of our most vulnerable children are educated - are met as quickly as possible.
This limited capital funding has become available by taking a more disciplined approach to managing my department’s capital budgets. Savings have been made by driving down the cost of new schools, shortening procurement times and challenging contractors to look for savings in all areas. These savings mean that more schools will benefit from the programme.
The PSBP will build on the progress we have already made in delivering a more efficient, faster, less bureaucratic approach to building schools. We are determined to reduce the wasteful processes of the past. That is why we have developed new baseline designs which will speed up the process and increase efficiencies and we are reducing the regulations and guidance governing school premises. This will encourage lower-cost build processes to be designed-in from the start.
I have previously expressed my strong support for the government’s agenda on reforming the PFI model and we are working closely with the Treasury to ensure the PSBP is aligned with this model in providing cost effective and more transparent delivery of services. Schools will have greater flexibility with soft facilities management services, such as cleaning, catering, security and some grounds maintenance being managed and controlled by schools themselves.
In addition to targeting spending on those schools which are in the worst condition, my priority in spending capital has been increasing the number of new school places in order to correct previous failures to meet that need. Since announcing the PSBP last July, the government has allocated £1.1 billion in additional funding to address the need for new school places.
Ends
Further details on the Priority School Building programme and a list of the schools that will be rebuilt, or have their condition needs met through the programme can be found on the school capital section of the Department for Education’s website.
Central newsdesk 020 7783 8300
General enquiries 0370 000 2288
Published: 24 May 2012


Construct-2 & Admob
1. Creating a Simple Construct 2 Game
In this chapter we will create a simple Construct 2 game for testing AdMob
ads.
2. Add Event Handlers
In this chapter, we will be adding event handlers for the buttons using
event sheets.
3. Admob Plugin
In this chapter we will learn how to download and install the AdMob plugin
for Construct 2.
4. Admob Event Sheet
In this chapter, we will insert our new Cordova AdmobFree object into our
project, and also create a new event sheet called Admob.
5. Complete Admob Functions
In this chapter, we will add 2 more functions to the Admob event sheet and
call those functions from Event sheet 1.
6. Exporting Project
In this chapter, we will learn how to export our project as a Cordova
project and then configure it for building an apk file.
7. Uploading to github
In this chapter, we will learn how to create a github account, install
git-scm and also how to create repositories and upload our project to
github.
8. Building an apk
In this chapter we will be looking at how to create a PhoneGap Build
account, creating a new app, linking the github repository and building an
apk file.
9. Android Emulator
In this chapter, we will learn how to download and install an Android
emulator called Nox Player, and how to transfer and install our apk files
on it.
10. Signing an apk
In this chapter we will learn how to create a release built for our apk.
11. What's Next
In this chapter we will explore your future options.
Back to the top


setActive ( ) {
Set the user's login status to active
.
Arguments
optional
|
String
user
If the Agent is idle
and has its User and Client already set, they do not need to be provided
again.
optional
|
String
client
If the Agent is idle
and has its User and Client already set, they do not need to be provided
again.
|
Boolean
rememberMe
A cookie management flag for browser clients. Whether the session credentials should be retained beyond the end of the session.


class SymbolHash
A subclass of Hash where all keys are converted into Symbols, and optionally, all String values are converted into Symbols.
Public Class Methods
@overload [](hash)
Creates a SymbolHash object from an existing Hash @example SymbolHash['x' => 1, :y => 2] # => #<SymbolHash:0x...> @param [Hash] hash the hash object @return [SymbolHash] a new SymbolHash from a hash object
@overload [](*list)
Creates a SymbolHash from an even list of keys and values @example SymbolHash[key1, value1, key2, value2, ...] @param [Array] list an even list of key followed by value @return [SymbolHash] a new SymbolHash object
# File lib/yard/core_ext/symbol_hash.rb, line 27 def self.[](*hsh) obj = new if hsh.size == 1 && hsh.first.is_a?(Hash) hsh.first.each {|k, v| obj[k] = v } else 0.step(hsh.size, 2) {|n| obj[hsh[n]] = hsh[n + 1] } end obj end
Creates a new SymbolHash object
@param [Boolean] symbolize_value converts any String values into Symbols
if this is set to +true+.
# File lib/yard/core_ext/symbol_hash.rb, line 8 def initialize(symbolize_value = true) @symbolize_value = symbolize_value end
Public Instance Methods
Accessed a symbolized key @param [#to_sym] key the key to access @return [Object] the value associated with the key
# File lib/yard/core_ext/symbol_hash.rb, line 48 def [](key) super(key.to_sym) end
Assigns a value to a symbolized key @param [#to_sym] key the key @param [Object] value the value to be assigned. If this is a String and
values are set to be symbolized, it will be converted into a Symbol.
# File lib/yard/core_ext/symbol_hash.rb, line 41 def []=(key, value) super(key.to_sym, value.instance_of?(String) && @symbolize_value ? value.to_sym : value) end
Deleted a key and value associated with it @param [#to_sym] key the key to delete @return [void]
# File lib/yard/core_ext/symbol_hash.rb, line 53 def delete(key) super(key.to_sym) end
Tests if a symbolized key exists @param [#to_sym] key the key to test @return [Boolean] whether the key exists
# File lib/yard/core_ext/symbol_hash.rb, line 58 def key?(key) super(key.to_sym) end
Merges the contents of another hash into a new SymbolHash object
@param [Hash] hash the hash of objects to copy @return [SymbolHash] a new SymbolHash containing the merged data
# File lib/yard/core_ext/symbol_hash.rb, line 73 def merge(hash) dup.merge!(hash) end
Updates the object with the contents of another Hash object. This method modifies the original SymbolHash object
@param [Hash] hash the hash object to copy the values from @return [SymbolHash] self
# File lib/yard/core_ext/symbol_hash.rb, line 66 def update(hash) hash.each {|k, v| self[k] = v }; self end


GoConvey - Go Testing Package: writing elegant tests
August 31, 2014
Introduction
Go comes with built-in unit test feature, and there are lots of third-party helper libraries before GoConvey was born. Unfortunately, none of them can help you write elegant test cases like GoConvey does, simple syntax and comfortable interface make you fall in love with writing unit tests.
Installation
go get github.com/smartystreets/goconvey
API Documentation
Please visit Go Walker.
Basic Usages
Write the code
Following code shows an example of basic four arithmetic(Add, subtract, multiply, divide):
package goconvey
import (
"errors"
)
func Add(a, b int) int {
return a + b
}
func Subtract(a, b int) int {
return a - b
}
func Multiply(a, b int) int {
return a * b
}
func Division(a, b int) (int, error) {
if b == 0 {
return 0, errors.New("can not divide by zero")
}
return a / b, nil
}
In the code above, we implemented 4 functions, now we will use GoConvey to write test cases for them:
package goconvey
import (
"testing"
. "github.com/smartystreets/goconvey/convey"
)
func TestAdd(t *testing.T) {
Convey("Add two numbers", t, func() {
So(Add(1, 2), ShouldEqual, 3)
})
}
func TestSubtract(t *testing.T) {
Convey("Subtract one from another", t, func() {
So(Subtract(1, 2), ShouldEqual, -1)
})
}
func TestMultiply(t *testing.T) {
Convey("Multiply two numbers", t, func() {
So(Multiply(3, 2), ShouldEqual, 6)
})
}
func TestDivision(t *testing.T) {
Convey("Divide one from another", t, func() {
Convey("Divide by non-zero number", func() {
num, err := Division(10, 2)
So(err, ShouldBeNil)
So(num, ShouldEqual, 5)
})
Convey("Divide by zero", func() {
_, err := Division(10, 0)
So(err, ShouldNotBeNil)
})
})
}
First of all, you should use the way that GoConvey team recommands to reduce redundant code: . "github.com/smartystreets/goconvey/convey"
.
Every test function has to start with Test
, such as TestAdd
, and needs a *testing.T
type argument.
To use GoConvey, every test case should use Convey
function to start with. Its first argument is a string
type argument; the second arguement usually is *testing.T
, which is the varible t
in this case; the third argument is a function with no argument and no return values(commonly use as closures).
You can have infinite nested Convey
statements to present relationships between test cases. In the example, TestDivision
function uses this way to shows the relations. Note that, only the out-most level need to pass varible t
, and no need for other levels.
Finally, you can use So
statement to test conditions. In this example, there are three different kinds of conditions: ShouldBeNil
, ShouldEqual
and ShouldNotBeNil
, each represents value should be nil, values should equal, and value should not be nil. For more built-in conditions, please see Documentation.
Run the tests
Now you can open the terminal and type go test -v
to run the tests. Because GoConvey compatibles with Go built-in unit test, we can directly use Go commands to execute tests.
Following content comes from the output of test cases of the example(Mac):
=== RUN TestAdd
Add two numbers ✔
1 assertion thus far
--- PASS: TestAdd (0.00 seconds)
=== RUN TestSubtract
Subtract one from another ✔
2 assertions thus far
--- PASS: TestSubtract (0.00 seconds)
=== RUN TestMultiply
Multiply two numbers ✔
3 assertions thus far
--- PASS: TestMultiply (0.00 seconds)
=== RUN TestDivision
Divide one from another
Divide by non-zero number ✔✔
Divide by zero ✔
6 assertions thus far
--- PASS: TestDivision (0.00 seconds)
PASS
ok github.com/Unknwon/go-rock-libraries-showcases/lectures/03-goconvey/class1/sample/goconvey 0.009s
As you can see, the output has very simple and understandable format, and test code are very elegant. But, is that all? Of course not! GoConvey has a very nice web interface to help automated testing as well.
Web Interface
To enable web interface of GoConvey, you need to execute goconvey
(install by go get
to $GOPATH/bin
), then open the browser and visit http://localhost:8080:
In the web interface, you can custom theme, see complete test results, browser notifications, etc.
Other features:
- Monitor code change and run tests
- Expressive DSL: http://localhost:8080/composer.html
- Coverage report: http://localhost:8080/reports/
- Stop monitoring a package
Summary
At this point, you should have a sense of how to use GoConvey to write awesome test cases, and how it makes boring work to be interesting.
Packages use GoConvey
To study in action, here is the list of packages that use GoConvey to write test cases:


Movie
public struct Movie : Codable
Undocumented
-
Undocumented
See moreDeclaration
Swift
public struct ReleaseDate : Codable
-
Undocumented
See moreDeclaration
Swift
public struct ReleaseDates : Codable
-
Undocumented
See moreDeclaration
Swift
public struct ProductionCountry : Codable
-
Undocumented
See moreDeclaration
Swift
public enum Status : String, Codable
-
Undocumented
Declaration
Swift
public let id: Int64
-
Undocumented
Declaration
Swift
public let adult: Bool
-
Undocumented
Declaration
Swift
public let budget: UInt?
-
Undocumented
Declaration
Swift
public let overview: String
-
Undocumented
Declaration
Swift
public let release: Date
-
Undocumented
Declaration
Swift
public let homepage: String?
-
Undocumented
Declaration
Swift
public let originalTitle: String
-
Undocumented
Declaration
Swift
public let originalLanguage: String
-
Undocumented
Declaration
Swift
public let title: String
-
Undocumented
Declaration
Swift
public let popularity: Double
-
Undocumented
Declaration
Swift
public let productionCountries: [ProductionCountry]?
-
Undocumented
Declaration
Swift
public let votes: UInt
-
Undocumented
Declaration
Swift
public let revenue: UInt?
-
Undocumented
Declaration
Swift
public let runtime: UInt?
-
Undocumented
Declaration
Swift
public let video: Bool
-
Undocumented
Declaration
Swift
public let averageVote: Double
-
Undocumented
Declaration
Swift
public let status: Status?
-
Undocumented
Declaration
Swift
public let tagline: String?
-
Undocumented
Declaration
Swift
public var releaseDates: [ReleaseDates]? { get }
-
Undocumented
Declaration
Swift
public var recommendations: [Movie]? { get }
-
Undocumented
Declaration
Swift
public var similar: [Movie]? { get }
-
Undocumented
Declaration
Swift
public func posterURL(size: TMDBImageSizeProtocol = TMDB.Utility.ImageSize.Poster.w154) -> URL?
-
Undocumented
Declaration
Swift
public func backdropURL(size: TMDBImageSizeProtocol = TMDB.Utility.ImageSize.Backdrop.w300) -> URL?


Prereq: (CS 245 or SE 212), (CS 246 or 247), (one of STAT 206, 230, 240); Computer Science students only. Antireq: CS 234, ECE 250, SE 240
Class Availability
Class Capacity
68
Enrollment Total
56
Available Seats
12
Description
Introduction to widely used and effective methods of data organization, focusing on data structures, their algorithms, and the performance of these algorithms. Specific topics include priority queues, sorting, dictionaries, data structures for text processing. [Note: Enrolment is restricted; see Note 1 above. Lab is not scheduled and students are expected to find time in open hours to complete their work. Offered: F,W,S]


These are hard-coded as instructed since the API change. Hence, having closed account placeholders and such functionality doen't seem to be relevant.
This seems to follow most of the functionality in the example app.
Logo
Channel
Program


Hogwild's blog
Home
All
About
RSS
About
Just say something about yourself.
(()
=>
console
.
log
(
'hello, world!'
))();


Use the API details to modify the metadata, plans and policies for the API. There is no need to follow the tabs in order, but you will need to fill out a minimum amount of data before the API can be published to the gateway. In particular, the Implementation and Plans tabs are important.
The Implementation tab is where you go to configure the details of the real API being managed. This is where you tell the gateway where the API is, so that we can actually invoke it at runtime.
The Definition tab allows you to configure an optional API definition document for your API. An API definition is a detailed description of the capabilities of your API (operation names, http verbs, data models). You can add a Swagger spec document as your API definition.
The Plans tab is used to configure which plans (defined in the organization) are available when a client wishes to create an API contract with this API. Optionally, an API can also be marked as Public. In that case, no API contract is required and the API can be invoked without sending an API Key.
The Developer Portal tab allows you to expose API versions via the Apiman Developer Portal, a separate user interface that is specifically designed to make it easy for external developers to discover and subscribe to APIs. You can choose from a wide array of options, including which plans you want to make available, and whether any of them require approval.
The Policies tab is where API-level policies can be configured. These policies will be enforced for all requests to the API, regardless of which user, Client App, or plan is being used.
The Contracts tab shows all of the registered consumers of the API. If the API is marked as Public, this tab will likely be empty.
The Endpoint tab will show you information about how to invoke the API once it is published to the API gateway.
The Metrics tab contains basic analytic information for the API. Use this tab to monitor the usage of the API.
The Change Log tab shows a history of all the changes made to the API. Essentially this is an audit log.


top
|
up
|
prev
|
next
BuildProjectRequest Struct
Definition at line
74
of
BuildServerMessage.hpp
Constructors
BuildProjectRequest
()
BuildProjectRequest
(JsonObject* messageObject)
Member Functions
void
DispatchTo
(
MessageHandler
& handler)
override
const
char
*
Id
()
const override
Constructor Details
BuildProjectRequest Constructor
cmajor::build::BuildProjectRequest::BuildProjectRequest()
Definition at line
102
of
BuildServerMessage.cpp
:
102
:
Message
<
ProjectToBuild
>
(
)
103
{
104
}
Declaration at line
76
of
BuildServerMessage.hpp
BuildProjectRequest Constructor
cmajor::build::BuildProjectRequest::BuildProjectRequest(JsonObject* messageObject)
Definition at line
106
of
BuildServerMessage.cpp
:
106
:
Message
<
ProjectToBuild
>
(
messageObject
)
107
{
108
}
Declaration at line
77
of
BuildServerMessage.hpp
Member Function Details
DispatchTo Member Function
void
cmajor::build::BuildProjectRequest::DispatchTo(
MessageHandler
& handler)
override
Definition at line
110
of
BuildServerMessage.cpp
:
111
{
112
handler
.
Handle
(
*
this
)
;
113
}
Declaration at line
79
of
BuildServerMessage.hpp
Calls:
cmajor::build::MessageHandler::Handle
Id Member Function
const
char
* cmajor::build::BuildProjectRequest::Id()
const override
Definition at line
78
of
BuildServerMessage.hpp
:
78
{
return
buildProjectRequestId
;
}
top
|
up
|
prev
|
next


--- layout: post esbma_id: 3102 title: The Multiple SQLite Problem date: 2014-05-16 12:00:00 keywords: sqlite ---
It's Google's fault. And Apple's fault.
No. Yes. Kinda. Not really.
If your app makes use of two separate instances of the SQLite library, you can end up with a corrupted SQLite data file.
On the SQLite website, section 2.2.1 of How to Corrupt an SQLite Database File is entitled "Multiple copies of SQLite linked into the same application", and says:
As pointed out in the previous paragraph, SQLite takes steps to work around the quirks of POSIX advisory locking. Part of that work-around involves keeping a global list (mutex protected) of open SQLite database files. But, if multiple copies of SQLite are linked into the same application, then there will be multiple instances of this global list. Database connections opened using one copy of the SQLite library will be unaware of database connections opened using the other copy, and will be unable to work around the POSIX advisory locking quirks. A close() operation on one connection might unknowingly clear the locks on a different database connection, leading to database corruption.
The scenario above sounds far-fetched. But the SQLite developers are aware of at least one commercial product that was released with exactly this bug. The vendor came to the SQLite developers seeking help in tracking down some infrequent database corruption issues they were seeing on Linux and Mac. The problem was eventually traced to the fact that the application was linking against two separate copies of SQLite. The solution was to change the application build procedures to link against just one copy of SQLite instead of two.
At its core, SQLite is written in C. It is plain-old-fashioned native/umanaged code. If you are accessing SQLite using C#, you are doing so through some kind of a wrapper. That wrapper is loading the SQLite library from somewhere. You may not know where. You probably don't [want to] care.
This is an abstraction. And it can leak. C# is putting some distance between you and the reality of what SQLite really is. That distance can somewhat increase the likelihood of you accidentally having two instances of the SQLite library without even knowing it.
Both iOS and Android contain an instance of SQLite as part of the basic operating system. This is a blessing. And a curse.
Built-in SQLite is nice because your app doesn't have to include it. This makes the size of your app smaller. It avoids the need to compile SQLite as part of your build process.
But the problem is that the OS has contributed one instance of the SQLite library that you can't eliminate. It's always there. The multiple SQLite problem cannot happen if only one SQLite is available to your app. Anybody or anything which adds one is risking a plurality.
Because Apple and Google do a terrible job of keeping it current.
iOS 7 ships with SQLite 3.7.13. That shipped in June of 2012.
Android ships with SQLite 3.7.11. That shipped in March of 2012.
Since Android users never update their devices, a large number of them are still running SQLite 3.7.4, which shipped in December of 2010. (Yes, I know the sweeping generalization in the previous sentence is unfair. I like Android a lot, but I think Google's management of the Android world has been bad enough that I'm entitled to a little crabbiness.)
If you are targeting Android or iOS and using the built-in SQLite library, you are missing out on at least TWO YEARS of excellent development work by DRH and his team. Current versions of SQLite are significantly faster, with many bug fixes, and lots of insanely cool new features. This is just one of the excellent reasons to bundle a current version of SQLite into your app instead of using the one in the OS.
And as soon as you do that, there are two instances in play. You and Apple/Google have collaborated to introduce the risk of database corruption.
AFAIK, no version of Windows includes a SQLite library. This is a blessing. And a curse. For all of the opposite reasons discussed above.
In general, building a mobile app for Windows (Phone or RT or whatever) means you have to include SQLite as part of the app. And when doing so, it certainly makes sense to just use the latest version.
And that introduces another reason somebody might want to use an application-private version of SQLite instead of the one built-in to iOS or Android. If you're building a cross-platform app, you probably want all your platforms using the same version of SQLite. Have fun explaining to your QA people that your app is built on SQLite 3.8.4 on Windows and 3.7.11 on Android and 3.7.13 on iOS.
BTW, it's not clear how or if Windows platforms suffer from the data corruption risk of the multiple SQLite problem. Given that the DRH explanation talks about workarounds for quirks in POSIX file locking, it seems likely that the situation on Windows is different in significant ways. Nonetheless, even if using multiple SQLite instances on Windows platforms is safe, it is still wasteful. And sad.
Mobile devices get lost or stolen. A significant portion of mobile app developers want their data encrypted on the device. And the SQLite instance built-in to iOS and Android is plain, with no support for encryption.
The usual solution to this problem is to use SQLCipher (open source, from Zetetic) or SEE (proprietary, from the authors of SQLite). Both of these are drop-in replacements for SQLite.
In other words, this is yet another reason the OS-provided SQLite library might not be sufficient.
SQLite can be compiled in a lot of different ways. Do you want the full-text-search feature? Do you want foreign keys to be default on or off? What do you want the default thread-safety mode to be? Do you need the column metadata feature? Do you need ICU for full Unicode support in collations? The list goes on and on.
Did Apple or Google compile SQLite with the exact set of build options your app needs? Maybe. Or maybe your app just needs to have its own.
Another way to get two SQLite instances is to add a component or library which includes one. Even if you don't know.
For example, the client side of Zumero (our mobile SQL sync product) needs to call SQLite. Should it bundle a SQLite library? Or should it always call the one in the mobile OS (when available)?
Some earlier versions of the Zumero client SDK included a SQLite instance in our Xamarin component builds. Because, why on earth would we want our code running against the archaic version of SQLite provided by Apple and Google?
 They called Zumero for sync. And they used Mono.Data.Sqlite for building their app.
Now we ship builds which contain no SQLite library instance, because it minimizes the likelihood of this kind of accident happening.
There are all kinds of libraries and components and SDKs out there which build on SQLite. Are they calling the instance provided by the OS? Or are they bundling one? Do you even know?
Knee-jerk reaction: Yes, absolutely.
Better answer: Certainly not.
App developers don't want to think about this stuff. It's a bit of esoterica that nobody cares about. Most people who started reading this blog entry gave up several paragraphs ago. The ones that are still here (both of you) are wondering why you are still reading when right now there are seven cable channels showing a rerun of Law and Order.
The multiple SQLite scenario is sounding less far-fetched all the time. SQLite is now one of the most widely deployed pieces of software in history. It is incredibly ubiquitous, and still growing. And people love to build abstractions on top of it.
This problem is going to get more and more common.
And it can have very significant consequences for end users.
The following requirements are very typical:
App developers want to be using a current version of SQLite (because DRH has actually been working for the last two years).
App developers want their SQLite data on the mobile device to be encrypted (because even grown-ups lose mobile devices).
App developers want to be using the same version of SQLite on all of their mobile app platforms (because it simplifies testing).
App developers want no risk of data corruption (because end users don't like that kind of thing).
App developers want to work with abstractions, also-known-as ORMs and sync tools, also-known-as things that makes their lives easier (because writing mobile apps is insanely expensive and it is important to reduce development costs).
App developers want to NOT have to think about anything in this blog entry (because they are paid to focus on their actual business, which is medicine or rental cars or construction, and it's 2014, so they shouldn't have to spend any time on the ramifications of quirky POSIX file locking).
Those requirements are not just typical, they are reasonable. To ask app developers to give up any of these things would be absurd.
And right now, there is NO WAY to satisfy all the requirements above. In the terminology of high school math, this is a system of equations with no solution.
The last several weeks of "the NuGet package is almost ready" are also due to some reasons I can't blame Apple or Google or POSIX for.
When I started working on SQLitePCL.raw, I didn't know nearly enough about MSBuild or NuGet. Anything involving native code with NuGet is pretty tricky. I've spent time climbing the learning curve. My particular way of learning new technologies is to write the code three times. The commit history on GitHub contains the whole story.
I want users of my SQLite PCL to have a great experience, so I'm spending [perhaps too much] time trying to find the sweetest subsets of the requirements above.
For example: C++/CX is actually pretty cool. I can build a single WP8 component DLL which is visible to C# while statically building SQLite itself inside. Fewer pieces. Fewer dependencies. Nice. But if anything else in the app needs direct access to SQLite, they'll have to include another instance of the library. Yuck.
Another example: I see [at least] three reasonable choices for iOS:
Use the SQLite provided by iOS. It's a shared library. Access it with P/Invoke, DllImport("sqlite3").
Bundle the latest SQLite. DllImport("__Internal"), and embed a sqlite3.a as a resource and use the MonoTouch LinkWith attribute.
Use the Xamarin SQLCipher component. DllImport("__Internal"), but don't bundle anything, relying on the presence of the SQLCipher component to make the link succeed.
Which one should the NuGet package assume that people want? How do people that prefer the others get a path that Just Works?
Soon. ;-)
"I don't know the key to success, but the key to failure is trying to please everybody." -- Bill Cosby