Introduction to Functional Programming in F# — Softwarepark This series of posts will introduce you to the world of functional programming (FP) in F#. Rather than start with theory or a formal definition, I thought that I'd start with a typical business problem and look at how we can use some of the functional pro...
TimeWarpEngineering/blazor-state Manage client side state in Blazor using MediatR pipeline. - TimeWarpEngineering/blazor-state
ASP.NET Blog | ASP.NET Core and Blazor updates in .NET Core 3.0 Release Candidate 1 .NET Core 3.0 Release Candidate 1 (RC1) is now available. This release contains only a handful of bug fixes and closely represents what we expect to release for .NET Core 3.0. Please see the release notes for additional details and known issues.
False or misleading claims There are laws in place to protect you from being misled about the products and services you buy. Businesses are not allowed to make statements that are incorrect or likely to create a false impression.
Events I'll be Attending in 2019 Here's what's known and I can talk about, there are always others on the back-burner or waiting to be announced. There's also always private events in various locations around the world.
xunit/xunit xUnit.net is a free, open source, community-focused unit testing tool for the .NET Framework. - xunit/xunit
Finalization implementation details | .NET Blog Years ago I wrote a document on making finalization scanning concurrent. At the time there was an internal team that was using finalization as a way to resurrect objects and putting them back in their cache. While we’ve always advised to folks that finali...
Legal Immigration: Last Week Tonight with John Oliver (HBO) John Oliver explains how our legal immigration system works, and how it often doesn’t. Connect with Last Week Tonight online... Subscribe to the Last Week Tonight YouTube channel for more almost news as it almost happens: www.youtube.com/lastweektonigh...
Bump mixin-deep from 1.3.1 to 1.3.2 by dependabot · Pull Request #1367 · buildo/react-components Bumps mixin-deep from 1.3.1 to 1.3.2. Commits 754f0c2 1.3.2 90ee1fa ensure keys are valid when mixing in values See full diff in compare view Maintainer changes This version was pushed to npm ...
BluestormDNA/ProjectPSX Experimental C# Playstation Emulator. Contribute to BluestormDNA/ProjectPSX development by creating an account on GitHub.
GC does not correctly evaluate the memory load on Linux · Issue #13371 · dotnet/runtime On Linux, when running in an unrestricted environment, the GC uses sysconf(SYSCONF_PAGES) * sysconf(_SC_PAGE_SIZE) to evaluate the total memory consumption of the system (https://github.com/dotnet/...
https://www.youtube.com/watch?v=r-TLSBdHe1A Performance clearly matters to users. For example, the most common software update on the AppStore is
beemdevelopment/Aegis A free, secure and open source app for Android to manage your 2-step verification tokens. - beemdevelopment/Aegis
Veldrid Veldrid is a low-level graphics library for .NET. It can be used to create high-performance 2D and 3D games, simulations, tools, and other graphical applications. Unlike most other .NET graphics libraries, Veldrid is designed to be portable, meaning it is...
Bye Bye Love Provided to YouTube by Elektra Records Bye Bye Love · The Cars The Cars ℗ 1987 Elektra/Asylum Records Bass Guitar: Benjamin Orr Additional Vocals: Benjamin Orr Drums: David Robinson Additional Vocals: David Robinson Guitar: Elliot Easton Additional...
BlazorHelp Website > Blog - Blazor Error Handling OwningComponentBase and EF Core Tracking Microsoft Visual Studio Blazor
http://phk.freebsd.dk/B-Heap/queue.html Would you believe me, if I claimed that an algorithm which has been on the books as "optimal" for 46 years, which have been analyzed in excruciating detail by geniuses like Knuth and taught in all CS courses in the world, can be optimized to run ten times...


Tell us about your event. This information is used to determine the eligibility of your event and what type of permit you require.
*All fields are requiredEvents on public property require approval from the local authority (local government, First Nation, Provincial or Federal government) to qualify for a Special Event Permit. A copy of this application may be reviewed by the local authority.
Please note that if you do not obtain written approval for permitted events on public property then your permit could be cancelled with no fees refunded
Unique requirements exist for the University of British Columbia. To learn more about application requirements at UBC visit the UBC Student Services website.
Please complete the rationale section below and state why you believe your event is of a provincial, national, or international significance.
The branch will review the rationale and may contact you for more information. Please see tool tip to the right of the text box for more information.


python_pachyderm.pfs_client module
# -*- coding: utf-8 -*- import collections import os from builtins import object from contextlib import contextmanager from .client.pfs.pfs_pb2 import * from .client.pfs.pfs_pb2_grpc import * BUFFER_SIZE = 3 * 1024 * 1024 # 3MB TODO: Base this on some grpc value class ExtractValueIterator(object): def __init__(self, r): self._iter = r def __next__(self): return next(self._iter).value def __iter__(self): return self def _commit_from(src, allow_just_repo=False): if src.__class__.__name__ == "Commit": return src elif type(src) in (tuple, list) and len(src) == 2: return Commit(repo=Repo(name=src[0]), id=src[1]) elif type(src) is str: repo_name, commit_id = src.split('/', 1) return Commit(repo=Repo(name=repo_name), id=commit_id) if not allow_just_repo: raise ValueError( "Commit should either be a sequence of [repo, commit_id] or a string in the form 'repo/branch/commit_id") return Commit(repo=Repo(name=src)) def _make_list(x): # if `x` is not iterable, put it in a list if isinstance(x, (str, bytes)) or not isinstance(x, collections.Iterable): x = [x] return x def _is_iterator(x): return hasattr(x, 'next') or hasattr(x, '__next__') class PfsClient(object): def __init__(self, host=None, port=None): """ Creates a client to connect to Pfs :param host: The pachd host. Default is 'localhost', which is used with `pachctl port-forward` :param port: The port to connect to. Default is 30650 """ # If a host or port is not specified, then try to set using environment variables or use the defaults. if host is None: host = os.environ.get('PACHD_SERVICE_HOST', 'localhost') if port is None: port = os.environ.get('PACHD_SERVICE_PORT_API_GRPC_PORT', '30650') self.channel = grpc.insecure_channel('{}:{}'.format(host, port)) self.stub = APIStub(self.channel) def create_repo(self, repo_name, description=None): """ Creates a new Repo object in pfs with the given name. Repos are the top level data object in pfs and should be used to store data of a similar type. For example rather than having a single Repo for an entire project you might have seperate Repos for logs, metrics, database dumps etc. :param repo_name: Name of the repo :param description: Repo description """ self.stub.CreateRepo(CreateRepoRequest(repo=Repo(name=repo_name), description=description)) def inspect_repo(self, repo_name): """ returns info about a specific Repo. :param repo_name: Name of the repo :return: A RepoInfo object """ return self.stub.InspectRepo(InspectRepoRequest(repo=Repo(name=repo_name))) def list_repo(self): """ Returns info about all Repos. :param provenance: Optional. Specifies a set of provenance repos where only repos which have ALL of the specified repos as provenance will be returned. :return: A list of RepoInfo objects """ x = self.stub.ListRepo(ListRepoRequest()) if hasattr(x, 'repo_info'): return x.repo_info return [] def delete_repo(self, repo_name=None, force=False, all=False): """ Deletes a repo and reclaims the storage space it was using. Note that as of 1.0 we do not reclaim the blocks that the Repo was referencing, this is because they may also be referenced by other Repos and deleting them would make those Repos inaccessible. This will be resolved in later versions. :param repo_name: The name of the repo :param force: if set to true, the repo will be removed regardless of errors. This argument should be used with care. :param all: Delete all repos """ if not all: if repo_name: self.stub.DeleteRepo(DeleteRepoRequest(repo=Repo(name=repo_name), force=force)) else: raise ValueError("Either a repo_name or all=True needs to be provided") else: if not repo_name: self.stub.DeleteRepo(DeleteRepoRequest(force=force, all=all)) else: raise ValueError("Cannot specify a repo_name if all=True") def start_commit(self, repo_name, branch=None, parent=None): """ Begins the process of committing data to a Repo. Once started you can write to the Commit with PutFile and when all the data has been written you must finish the Commit with FinishCommit. NOTE, data is not persisted until FinishCommit is called. :param repo_name: The name of the repo :param branch: is a more convenient way to build linear chains of commits. When a commit is started with a non empty branch the value of branch becomes an alias for the created Commit. This enables a more intuitive access pattern. When the commit is started on a branch the previous head of the branch is used as the parent of the commit. :param parent: specifies the parent Commit, upon creation the new Commit will appear identical to the parent Commit, data can safely be added to the new commit without affecting the contents of the parent Commit. You may pass "" as parentCommit in which case the new Commit will have no parent and will initially appear empty. :return: Commit object """ return self.stub.StartCommit(StartCommitRequest(parent=Commit(repo=Repo(name=repo_name), id=parent), branch=branch)) def finish_commit(self, commit): """ Ends the process of committing data to a Repo and persists the Commit.  " "Trying to finish it (Currently a commit can't be cancelled)") raise e finally: self.finish_commit(commit) def inspect_commit(self, commit): """ returns info about a specific Commit. :param commit: A tuple or string representing the commit :return: CommitInfo object """ return self.stub.InspectCommit(InspectCommitRequest(commit=_commit_from(commit))) def provenances_for_repo(self, repo_name): provenances = {} commits = self.list_commit(repo_name) sorted_commits = [x[0] for x in sorted([(c.commit.id, c.finished.seconds) for c in commits], key=lambda x: x[1])] for c in sorted_commits: for p in c.provenance: provenances[p.id] = c.commit.id return provenances def list_commit(self, repo_name, to_commit=None, from_commit=None, number=0): """ Lists commits. :param repo_name: If only `repo_name` is given, all commits in the repo are returned. :param to_commit: optional. only the ancestors of `to`, including `to` itself, are considered. :param from_commit: optional. only the descendents of `from`, including `from` itself, are considered. :param number: optional. determines how many commits are returned. If `number` is 0, all commits that match the aforementioned criteria are returned. :return: A list of CommitInfo objects """ req = ListCommitRequest(repo=Repo(name=repo_name), number=number) if to_commit is not None: req.to.CopyFrom(_commit_from(to_commit)) if from_commit is not None: getattr(req, 'from').CopyFrom(_commit_from(from_commit)) x = self.stub.ListCommit(req) if hasattr(x, 'commit_info'): return x.commit_info return [] def delete_commit(self, commit): """ deletes a commit. Note it is currently not implemented. :param commit: A tuple or string representing the commit """ self.stub.DeleteCommit(DeleteCommitRequest(commit=_commit_from(commit))) def flush_commit(self, commits, repos=tuple()): """ blocks until all of the commits which have a set of commits as provenance have finished. For commits to be considered they must have all of the specified commits as provenance. This in effect waits for all of the jobs that are triggered by a set of commits to complete. It returns an error if any of the commits it's waiting on are cancelled due to one of the jobs encountering an error during runtime. Note that it's never necessary to call FlushCommit to run jobs, they'll run no matter what, FlushCommit just allows you to wait for them to complete and see their output once they do. :param commits: A commit or a list of commits to wait on :param repos: Optional. Only the commits up to and including those repos will be considered, otherwise all repos are considered. :return: An iterator of CommitInfo objects """ return self.stub.FlushCommit(FlushCommitRequest(commit=[_commit_from(c) for c in commits], to_repo=[Repo(name=r) for r in repos])) def subscribe_commit(self, repo_name, branch, from_commit_id=None): """ SubscribeCommit is like ListCommit but it keeps listening for commits as they come in. :param repo_name: Name of the repo :param branch: Branch to subscribe to :param from_commit_id: Optional. only commits created since this commit are returned :return: Iterator of Commit objects """ repo = Repo(name=repo_name) req = SubscribeCommitRequest(repo=repo, branch=branch) if from_commit_id is not None: getattr(req, 'from').CopyFrom(Commit(repo=repo, id=from_commit_id)) return self.stub.SubscribeCommit(req) def list_branch(self, repo_name): """ lists the active branches on a Repo :param repo_name: The name of the repo :return: A list of Branch objects """ x = self.stub.ListBranch(ListBranchRequest(repo=Repo(name=repo_name))) if hasattr(x, 'branch_info'): return x.branch_info return [] def set_branch(self, commit, branch_name): """ sets a commit and its ancestors as a branch :param commit: A tuple or string representing the commit :param branch_name: The name for the branch to set """ self.stub.SetBranch(SetBranchRequest(commit=_commit_from(commit), branch=branch_name)) def delete_branch(self, repo_name, branch_name): """ deletes a branch, but leaves the commits themselves intact. In other words, those commits can still be accessed via commit IDs and other branches they happen to be on. :param repo_name: The name of the repo :param branch_name: The name of the branch to delete """ self.stub.DeleteBranch(DeleteBranchRequest(repo=Repo(name=repo_name), branch=branch_name)) def put_file_bytes(self, commit, path, value, delimiter=NONE, target_file_datums=0, target_file_bytes=0): """ Uploads a binary bytes array as file(s) in a certain path :param commit: A tuple or string representing the commit :param path: Path in the repo the file(s) will be written to :param value: The data bytes array, or an iterator returning chunked byte arrays :param delimiter: Optional. causes data to be broken up into separate files with `path` as a prefix. :param target_file_datums: Optional. specifies the target number of datums in each written file it may be lower if data does not split evenly, but will never be higher, unless the value is 0. :param target_file_bytes: specifies the target number of bytes in each written file, files may have more or fewer bytes than the target. """ if _is_iterator(value): def _wrap(v): for x in v: yield PutFileRequest(file=File(commit=_commit_from(commit), path=path), value=x, delimiter=delimiter, target_file_datums=target_file_datums, target_file_bytes=target_file_bytes) self.stub.PutFile(_wrap(value)) return def _blocks(v): for i in range(0, len(v), BUFFER_SIZE): yield PutFileRequest(file=File(commit=_commit_from(commit), path=path), value=v[i:i + BUFFER_SIZE], delimiter=delimiter, target_file_datums=target_file_datums, target_file_bytes=target_file_bytes) self.stub.PutFile(_blocks(value)) def put_file_url(self, commit, path, url, recursive=False): """ puts a file using the content found at a URL. The URL is sent to the server which performs the request. :param commit: A tuple or string representing the commit :param path: The path to the file :param url: The url to download :param recursive: allow for recursive scraping of some types URLs for example on s3:// urls. """ self.stub.PutFile(iter([PutFileRequest(file=File(commit=_commit_from(commit), path=path), url=url, recursive=recursive)])) def get_file(self, commit, path, offset_bytes=0, size_bytes=0, extract_value=True): """ returns the contents of a file at a specific Commit. :param commit: A tuple or string representing the commit :param path: The path of the file :param offset_bytes: Optional. specifies a number of bytes that should be skipped in the beginning of the file. :param size_bytes: Optional. limits the total amount of data returned, note you will get fewer bytes than size if you pass a value larger than the size of the file. If size is set to 0 then all of the data will be returned. :param extract_value: If True, then an ExtractValueIterator will be return, which will iterate over the bytes of the file. If False, then the Protobuf response iterator will return :return: An iterator over the file or an iterator over the protobuf responses """ r = self.stub.GetFile(GetFileRequest(file=File(commit=_commit_from(commit), path=path), offset_bytes=offset_bytes, size_bytes=size_bytes)) if extract_value: return ExtractValueIterator(r) return r def get_files(self, commit, paths, recursive=False): """ returns the contents of a list of files at a specific Commit. :param commit: A tuple or string representing the commit :param paths: A list of paths to retrieve :param recursive: If True, will go into each directory in the list recursively :return: A dictionary of file paths and data """ filtered_file_infos = [] for path in paths: fi = self.inspect_file(commit, path) if fi.file_type == FILE: filtered_file_infos.append(fi) else: filtered_file_infos += self.list_file(commit, path, recursive=recursive) filtered_paths = [fi.file.path for fi in filtered_file_infos if fi.file_type == FILE] return {path: b''.join(self.get_file(commit, path)) for path in filtered_paths} def inspect_file(self, commit, path): """ returns info about a specific file. :param commit: A tuple or string representing the commit :param path: Path to file :return: A FileInfo object """ return self.stub.InspectFile(InspectFileRequest(file=File(commit=_commit_from(commit), path=path))) def list_file(self, commit, path, recursive=False): """ Lists the files in a directory :param commit: A tuple or string representing the commit :param path: The path to the directory :param recursive: If True, continue listing the files for sub-directories :return: A list of FileInfo objects """ file_infos = self.stub.ListFile(ListFileRequest(file=File(commit=_commit_from(commit), path=path))).file_info if recursive: dirs = [f for f in file_infos if f.file_type == DIR] files = [f for f in file_infos if f.file_type == FILE] return sum([self.list_file(commit, d.file.path, recursive) for d in dirs], files) return list(file_infos) def glob_file(self, commit, pattern): """ ? :param commit: :param pattern: :return: A list of FileInfo objects """ r = self.stub.GlobFile(GlobFileRequest(commit=_commit_from(commit), pattern=pattern)) if hasattr(r, 'file_info'): return r.file_info return [] def delete_file(self, commit, path): """ deletes a file from a Commit. DeleteFile leaves a tombstone in the Commit, assuming the file isn't written to later attempting to get the file from the finished commit will result in not found error. The file will of course remain intact in the Commit's parent. :param commit: A tuple or string representing the commit :param path: The path to the file """ self.stub.DeleteFile(DeleteFileRequest(file=File(commit=_commit_from(commit), path=path))) def delete_all(self): self.stub.DeleteAll(google_dot_protobuf_dot_empty__pb2.Empty())
Module variables
var BUFFER_SIZE
var CSV
var DESCRIPTOR
var DIR
var FILE
var FINISHED
var JSON
var LINE
var NONE
var READY
var RESERVED
var SQL
var STARTED
Classes
class Block
A ProtocolMessage
Ancestors (in MRO)
- Block
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var HASH_FIELD_NUMBER
class BlockRef
A ProtocolMessage
Ancestors (in MRO)
- BlockRef
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BLOCK_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var RANGE_FIELD_NUMBER
class Branch
A ProtocolMessage
Ancestors (in MRO)
- Branch
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var NAME_FIELD_NUMBER
var REPO_FIELD_NUMBER
class BranchInfo
A ProtocolMessage
Ancestors (in MRO)
- BranchInfo
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BRANCH_FIELD_NUMBER
var DESCRIPTOR
var DIRECT_PROVENANCE_FIELD_NUMBER
var Extensions
var HEAD_FIELD_NUMBER
var NAME_FIELD_NUMBER
var PROVENANCE_FIELD_NUMBER
var SUBVENANCE_FIELD_NUMBER
class BranchInfos
A ProtocolMessage
Ancestors (in MRO)
- BranchInfos
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BRANCH_INFO_FIELD_NUMBER
var DESCRIPTOR
var Extensions
class BuildCommitRequest
A ProtocolMessage
Ancestors (in MRO)
- BuildCommitRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BRANCH_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var ID_FIELD_NUMBER
var PARENT_FIELD_NUMBER
var PROVENANCE_FIELD_NUMBER
var TREE_FIELD_NUMBER
class ByteRange
A ProtocolMessage
Ancestors (in MRO)
- ByteRange
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var LOWER_FIELD_NUMBER
var UPPER_FIELD_NUMBER
class CheckObjectRequest
A ProtocolMessage
Ancestors (in MRO)
- CheckObjectRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var OBJECT_FIELD_NUMBER
class CheckObjectResponse
A ProtocolMessage
Ancestors (in MRO)
- CheckObjectResponse
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var EXISTS_FIELD_NUMBER
var Extensions
class Commit
A ProtocolMessage
Ancestors (in MRO)
- Commit
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var ID_FIELD_NUMBER
var REPO_FIELD_NUMBER
class CommitInfo
A ProtocolMessage
Ancestors (in MRO)
- CommitInfo
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BRANCH_PROVENANCE_FIELD_NUMBER
var CHILD_COMMITS_FIELD_NUMBER
var COMMIT_FIELD_NUMBER
var DATUMS_FIELD_NUMBER
var DESCRIPTION_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var FINISHED_FIELD_NUMBER
var PARENT_COMMIT_FIELD_NUMBER
var PROVENANCE_FIELD_NUMBER
var READY_PROVENANCE_FIELD_NUMBER
var SIZE_BYTES_FIELD_NUMBER
var STARTED_FIELD_NUMBER
var SUBVENANCE_FIELD_NUMBER
var TREES_FIELD_NUMBER
var TREE_FIELD_NUMBER
class CommitInfos
A ProtocolMessage
Ancestors (in MRO)
- CommitInfos
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var COMMIT_INFO_FIELD_NUMBER
var DESCRIPTOR
var Extensions
class CommitRange
A ProtocolMessage
Ancestors (in MRO)
- CommitRange
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var LOWER_FIELD_NUMBER
var UPPER_FIELD_NUMBER
class CopyFileRequest
A ProtocolMessage
Ancestors (in MRO)
- CopyFileRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var DST_FIELD_NUMBER
var Extensions
var OVERWRITE_FIELD_NUMBER
var SRC_FIELD_NUMBER
class CreateBranchRequest
A ProtocolMessage
Ancestors (in MRO)
- CreateBranchRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BRANCH_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var HEAD_FIELD_NUMBER
var PROVENANCE_FIELD_NUMBER
var S_BRANCH_FIELD_NUMBER
class CreateRepoRequest
A ProtocolMessage
Ancestors (in MRO)
- CreateRepoRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTION_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var REPO_FIELD_NUMBER
var UPDATE_FIELD_NUMBER
class DeleteBranchRequest
A ProtocolMessage
Ancestors (in MRO)
- DeleteBranchRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BRANCH_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var FORCE_FIELD_NUMBER
class DeleteCommitRequest
A ProtocolMessage
Ancestors (in MRO)
- DeleteCommitRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var COMMIT_FIELD_NUMBER
var DESCRIPTOR
var Extensions
class DeleteFileRequest
A ProtocolMessage
Ancestors (in MRO)
- DeleteFileRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var FILE_FIELD_NUMBER
class DeleteObjectsRequest
A ProtocolMessage
Ancestors (in MRO)
- DeleteObjectsRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var OBJECTS_FIELD_NUMBER
class DeleteObjectsResponse
A ProtocolMessage
Ancestors (in MRO)
- DeleteObjectsResponse
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
class DeleteRepoRequest
A ProtocolMessage
Ancestors (in MRO)
- DeleteRepoRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var ALL_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var FORCE_FIELD_NUMBER
var REPO_FIELD_NUMBER
class DeleteTagsRequest
A ProtocolMessage
Ancestors (in MRO)
- DeleteTagsRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var TAGS_FIELD_NUMBER
class DeleteTagsResponse
A ProtocolMessage
Ancestors (in MRO)
- DeleteTagsResponse
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
class DiffFileRequest
A ProtocolMessage
Ancestors (in MRO)
- DiffFileRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var NEW_FILE_FIELD_NUMBER
var OLD_FILE_FIELD_NUMBER
var SHALLOW_FIELD_NUMBER
class DiffFileResponse
A ProtocolMessage
Ancestors (in MRO)
- DiffFileResponse
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var NEW_FILES_FIELD_NUMBER
var OLD_FILES_FIELD_NUMBER
class ExtractValueIterator
class ExtractValueIterator(object): def __init__(self, r): self._iter = r def __next__(self): return next(self._iter).value def __iter__(self): return self
Ancestors (in MRO)
- ExtractValueIterator
- builtins.object
Static methods
def __init__(
self, r)
Initialize self. See help(type(self)) for accurate signature.
def __init__(self, r): self._iter = r
class File
A ProtocolMessage
Ancestors (in MRO)
- File
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var COMMIT_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var PATH_FIELD_NUMBER
class FileInfo
A ProtocolMessage
Ancestors (in MRO)
- FileInfo
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BLOCKREFS_FIELD_NUMBER
var CHILDREN_FIELD_NUMBER
var COMMITTED_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var FILE_FIELD_NUMBER
var FILE_TYPE_FIELD_NUMBER
var HASH_FIELD_NUMBER
var OBJECTS_FIELD_NUMBER
var SIZE_BYTES_FIELD_NUMBER
class FileInfos
A ProtocolMessage
Ancestors (in MRO)
- FileInfos
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var FILE_INFO_FIELD_NUMBER
class FinishCommitRequest
A ProtocolMessage
Ancestors (in MRO)
- FinishCommitRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var COMMIT_FIELD_NUMBER
var DATUMS_FIELD_NUMBER
var DESCRIPTION_FIELD_NUMBER
var DESCRIPTOR
var EMPTY_FIELD_NUMBER
var Extensions
var SIZE_BYTES_FIELD_NUMBER
var TREES_FIELD_NUMBER
var TREE_FIELD_NUMBER
class FlushCommitRequest
A ProtocolMessage
Ancestors (in MRO)
- FlushCommitRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var COMMITS_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var TO_REPOS_FIELD_NUMBER
class GetBlocksRequest
A ProtocolMessage
Ancestors (in MRO)
- GetBlocksRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BLOCKREFS_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var OFFSET_BYTES_FIELD_NUMBER
var SIZE_BYTES_FIELD_NUMBER
var TOTAL_SIZE_FIELD_NUMBER
class GetFileRequest
A ProtocolMessage
Ancestors (in MRO)
- GetFileRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var FILE_FIELD_NUMBER
var OFFSET_BYTES_FIELD_NUMBER
var SIZE_BYTES_FIELD_NUMBER
class GetObjectsRequest
A ProtocolMessage
Ancestors (in MRO)
- GetObjectsRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var OBJECTS_FIELD_NUMBER
var OFFSET_BYTES_FIELD_NUMBER
var SIZE_BYTES_FIELD_NUMBER
var TOTAL_SIZE_FIELD_NUMBER
class GlobFileRequest
A ProtocolMessage
Ancestors (in MRO)
- GlobFileRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var COMMIT_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var PATTERN_FIELD_NUMBER
class InspectBranchRequest
A ProtocolMessage
Ancestors (in MRO)
- InspectBranchRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BRANCH_FIELD_NUMBER
var DESCRIPTOR
var Extensions
class InspectCommitRequest
A ProtocolMessage
Ancestors (in MRO)
- InspectCommitRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BLOCK_STATE_FIELD_NUMBER
var COMMIT_FIELD_NUMBER
var DESCRIPTOR
var Extensions
class InspectFileRequest
A ProtocolMessage
Ancestors (in MRO)
- InspectFileRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var FILE_FIELD_NUMBER
class InspectRepoRequest
A ProtocolMessage
Ancestors (in MRO)
- InspectRepoRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var REPO_FIELD_NUMBER
class ListBranchRequest
A ProtocolMessage
Ancestors (in MRO)
- ListBranchRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var REPO_FIELD_NUMBER
class ListCommitRequest
A ProtocolMessage
Ancestors (in MRO)
- ListCommitRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var FROM_FIELD_NUMBER
var NUMBER_FIELD_NUMBER
var REPO_FIELD_NUMBER
var TO_FIELD_NUMBER
class ListFileRequest
A ProtocolMessage
Ancestors (in MRO)
- ListFileRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var FILE_FIELD_NUMBER
var FULL_FIELD_NUMBER
var HISTORY_FIELD_NUMBER
class ListObjectsRequest
A ProtocolMessage
Ancestors (in MRO)
- ListObjectsRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
class ListRepoRequest
A ProtocolMessage
Ancestors (in MRO)
- ListRepoRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
class ListRepoResponse
A ProtocolMessage
Ancestors (in MRO)
- ListRepoResponse
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var REPO_INFO_FIELD_NUMBER
class ListTagsRequest
A ProtocolMessage
Ancestors (in MRO)
- ListTagsRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var INCLUDE_OBJECT_FIELD_NUMBER
var PREFIX_FIELD_NUMBER
class ListTagsResponse
A ProtocolMessage
Ancestors (in MRO)
- ListTagsResponse
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var OBJECT_FIELD_NUMBER
var TAG_FIELD_NUMBER
class Object
A ProtocolMessage
Ancestors (in MRO)
- Object
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var HASH_FIELD_NUMBER
class ObjectIndex
A ProtocolMessage
Ancestors (in MRO)
- ObjectIndex
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var OBJECTS_FIELD_NUMBER
var ObjectsEntry
var TAGS_FIELD_NUMBER
var TagsEntry
class ObjectInfo
A ProtocolMessage
Ancestors (in MRO)
- ObjectInfo
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BLOCK_REF_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var OBJECT_FIELD_NUMBER
class Objects
A ProtocolMessage
Ancestors (in MRO)
- Objects
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var OBJECTS_FIELD_NUMBER
class OverwriteIndex
A ProtocolMessage
Ancestors (in MRO)
- OverwriteIndex
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var INDEX_FIELD_NUMBER
class PfsClient
class PfsClient(object): def __init__(self, host=None, port=None): """ Creates a client to connect to Pfs :param host: The pachd host. Default is 'localhost', which is used with `pachctl port-forward` :param port: The port to connect to. Default is 30650 """ # If a host or port is not specified, then try to set using environment variables or use the defaults. if host is None: host = os.environ.get('PACHD_SERVICE_HOST', 'localhost') if port is None: port = os.environ.get('PACHD_SERVICE_PORT_API_GRPC_PORT', '30650') self.channel = grpc.insecure_channel('{}:{}'.format(host, port)) self.stub = APIStub(self.channel) def create_repo(self, repo_name, description=None): """ Creates a new Repo object in pfs with the given name. Repos are the top level data object in pfs and should be used to store data of a similar type. For example rather than having a single Repo for an entire project you might have seperate Repos for logs, metrics, database dumps etc. :param repo_name: Name of the repo :param description: Repo description """ self.stub.CreateRepo(CreateRepoRequest(repo=Repo(name=repo_name), description=description)) def inspect_repo(self, repo_name): """ returns info about a specific Repo. :param repo_name: Name of the repo :return: A RepoInfo object """ return self.stub.InspectRepo(InspectRepoRequest(repo=Repo(name=repo_name))) def list_repo(self): """ Returns info about all Repos. :param provenance: Optional. Specifies a set of provenance repos where only repos which have ALL of the specified repos as provenance will be returned. :return: A list of RepoInfo objects """ x = self.stub.ListRepo(ListRepoRequest()) if hasattr(x, 'repo_info'): return x.repo_info return [] def delete_repo(self, repo_name=None, force=False, all=False): """ Deletes a repo and reclaims the storage space it was using. Note that as of 1.0 we do not reclaim the blocks that the Repo was referencing, this is because they may also be referenced by other Repos and deleting them would make those Repos inaccessible. This will be resolved in later versions. :param repo_name: The name of the repo :param force: if set to true, the repo will be removed regardless of errors. This argument should be used with care. :param all: Delete all repos """ if not all: if repo_name: self.stub.DeleteRepo(DeleteRepoRequest(repo=Repo(name=repo_name), force=force)) else: raise ValueError("Either a repo_name or all=True needs to be provided") else: if not repo_name: self.stub.DeleteRepo(DeleteRepoRequest(force=force, all=all)) else: raise ValueError("Cannot specify a repo_name if all=True") def start_commit(self, repo_name, branch=None, parent=None): """ Begins the process of committing data to a Repo. Once started you can write to the Commit with PutFile and when all the data has been written you must finish the Commit with FinishCommit. NOTE, data is not persisted until FinishCommit is called. :param repo_name: The name of the repo :param branch: is a more convenient way to build linear chains of commits. When a commit is started with a non empty branch the value of branch becomes an alias for the created Commit. This enables a more intuitive access pattern. When the commit is started on a branch the previous head of the branch is used as the parent of the commit. :param parent: specifies the parent Commit, upon creation the new Commit will appear identical to the parent Commit, data can safely be added to the new commit without affecting the contents of the parent Commit. You may pass "" as parentCommit in which case the new Commit will have no parent and will initially appear empty. :return: Commit object """ return self.stub.StartCommit(StartCommitRequest(parent=Commit(repo=Repo(name=repo_name), id=parent), branch=branch)) def finish_commit(self, commit): """ Ends the process of committing data to a Repo and persists the Commit.  " "Trying to finish it (Currently a commit can't be cancelled)") raise e finally: self.finish_commit(commit) def inspect_commit(self, commit): """ returns info about a specific Commit. :param commit: A tuple or string representing the commit :return: CommitInfo object """ return self.stub.InspectCommit(InspectCommitRequest(commit=_commit_from(commit))) def provenances_for_repo(self, repo_name): provenances = {} commits = self.list_commit(repo_name) sorted_commits = [x[0] for x in sorted([(c.commit.id, c.finished.seconds) for c in commits], key=lambda x: x[1])] for c in sorted_commits: for p in c.provenance: provenances[p.id] = c.commit.id return provenances def list_commit(self, repo_name, to_commit=None, from_commit=None, number=0): """ Lists commits. :param repo_name: If only `repo_name` is given, all commits in the repo are returned. :param to_commit: optional. only the ancestors of `to`, including `to` itself, are considered. :param from_commit: optional. only the descendents of `from`, including `from` itself, are considered. :param number: optional. determines how many commits are returned. If `number` is 0, all commits that match the aforementioned criteria are returned. :return: A list of CommitInfo objects """ req = ListCommitRequest(repo=Repo(name=repo_name), number=number) if to_commit is not None: req.to.CopyFrom(_commit_from(to_commit)) if from_commit is not None: getattr(req, 'from').CopyFrom(_commit_from(from_commit)) x = self.stub.ListCommit(req) if hasattr(x, 'commit_info'): return x.commit_info return [] def delete_commit(self, commit): """ deletes a commit. Note it is currently not implemented. :param commit: A tuple or string representing the commit """ self.stub.DeleteCommit(DeleteCommitRequest(commit=_commit_from(commit))) def flush_commit(self, commits, repos=tuple()): """ blocks until all of the commits which have a set of commits as provenance have finished. For commits to be considered they must have all of the specified commits as provenance. This in effect waits for all of the jobs that are triggered by a set of commits to complete. It returns an error if any of the commits it's waiting on are cancelled due to one of the jobs encountering an error during runtime. Note that it's never necessary to call FlushCommit to run jobs, they'll run no matter what, FlushCommit just allows you to wait for them to complete and see their output once they do. :param commits: A commit or a list of commits to wait on :param repos: Optional. Only the commits up to and including those repos will be considered, otherwise all repos are considered. :return: An iterator of CommitInfo objects """ return self.stub.FlushCommit(FlushCommitRequest(commit=[_commit_from(c) for c in commits], to_repo=[Repo(name=r) for r in repos])) def subscribe_commit(self, repo_name, branch, from_commit_id=None): """ SubscribeCommit is like ListCommit but it keeps listening for commits as they come in. :param repo_name: Name of the repo :param branch: Branch to subscribe to :param from_commit_id: Optional. only commits created since this commit are returned :return: Iterator of Commit objects """ repo = Repo(name=repo_name) req = SubscribeCommitRequest(repo=repo, branch=branch) if from_commit_id is not None: getattr(req, 'from').CopyFrom(Commit(repo=repo, id=from_commit_id)) return self.stub.SubscribeCommit(req) def list_branch(self, repo_name): """ lists the active branches on a Repo :param repo_name: The name of the repo :return: A list of Branch objects """ x = self.stub.ListBranch(ListBranchRequest(repo=Repo(name=repo_name))) if hasattr(x, 'branch_info'): return x.branch_info return [] def set_branch(self, commit, branch_name): """ sets a commit and its ancestors as a branch :param commit: A tuple or string representing the commit :param branch_name: The name for the branch to set """ self.stub.SetBranch(SetBranchRequest(commit=_commit_from(commit), branch=branch_name)) def delete_branch(self, repo_name, branch_name): """ deletes a branch, but leaves the commits themselves intact. In other words, those commits can still be accessed via commit IDs and other branches they happen to be on. :param repo_name: The name of the repo :param branch_name: The name of the branch to delete """ self.stub.DeleteBranch(DeleteBranchRequest(repo=Repo(name=repo_name), branch=branch_name)) def put_file_bytes(self, commit, path, value, delimiter=NONE, target_file_datums=0, target_file_bytes=0): """ Uploads a binary bytes array as file(s) in a certain path :param commit: A tuple or string representing the commit :param path: Path in the repo the file(s) will be written to :param value: The data bytes array, or an iterator returning chunked byte arrays :param delimiter: Optional. causes data to be broken up into separate files with `path` as a prefix. :param target_file_datums: Optional. specifies the target number of datums in each written file it may be lower if data does not split evenly, but will never be higher, unless the value is 0. :param target_file_bytes: specifies the target number of bytes in each written file, files may have more or fewer bytes than the target. """ if _is_iterator(value): def _wrap(v): for x in v: yield PutFileRequest(file=File(commit=_commit_from(commit), path=path), value=x, delimiter=delimiter, target_file_datums=target_file_datums, target_file_bytes=target_file_bytes) self.stub.PutFile(_wrap(value)) return def _blocks(v): for i in range(0, len(v), BUFFER_SIZE): yield PutFileRequest(file=File(commit=_commit_from(commit), path=path), value=v[i:i + BUFFER_SIZE], delimiter=delimiter, target_file_datums=target_file_datums, target_file_bytes=target_file_bytes) self.stub.PutFile(_blocks(value)) def put_file_url(self, commit, path, url, recursive=False): """ puts a file using the content found at a URL. The URL is sent to the server which performs the request. :param commit: A tuple or string representing the commit :param path: The path to the file :param url: The url to download :param recursive: allow for recursive scraping of some types URLs for example on s3:// urls. """ self.stub.PutFile(iter([PutFileRequest(file=File(commit=_commit_from(commit), path=path), url=url, recursive=recursive)])) def get_file(self, commit, path, offset_bytes=0, size_bytes=0, extract_value=True): """ returns the contents of a file at a specific Commit. :param commit: A tuple or string representing the commit :param path: The path of the file :param offset_bytes: Optional. specifies a number of bytes that should be skipped in the beginning of the file. :param size_bytes: Optional. limits the total amount of data returned, note you will get fewer bytes than size if you pass a value larger than the size of the file. If size is set to 0 then all of the data will be returned. :param extract_value: If True, then an ExtractValueIterator will be return, which will iterate over the bytes of the file. If False, then the Protobuf response iterator will return :return: An iterator over the file or an iterator over the protobuf responses """ r = self.stub.GetFile(GetFileRequest(file=File(commit=_commit_from(commit), path=path), offset_bytes=offset_bytes, size_bytes=size_bytes)) if extract_value: return ExtractValueIterator(r) return r def get_files(self, commit, paths, recursive=False): """ returns the contents of a list of files at a specific Commit. :param commit: A tuple or string representing the commit :param paths: A list of paths to retrieve :param recursive: If True, will go into each directory in the list recursively :return: A dictionary of file paths and data """ filtered_file_infos = [] for path in paths: fi = self.inspect_file(commit, path) if fi.file_type == FILE: filtered_file_infos.append(fi) else: filtered_file_infos += self.list_file(commit, path, recursive=recursive) filtered_paths = [fi.file.path for fi in filtered_file_infos if fi.file_type == FILE] return {path: b''.join(self.get_file(commit, path)) for path in filtered_paths} def inspect_file(self, commit, path): """ returns info about a specific file. :param commit: A tuple or string representing the commit :param path: Path to file :return: A FileInfo object """ return self.stub.InspectFile(InspectFileRequest(file=File(commit=_commit_from(commit), path=path))) def list_file(self, commit, path, recursive=False): """ Lists the files in a directory :param commit: A tuple or string representing the commit :param path: The path to the directory :param recursive: If True, continue listing the files for sub-directories :return: A list of FileInfo objects """ file_infos = self.stub.ListFile(ListFileRequest(file=File(commit=_commit_from(commit), path=path))).file_info if recursive: dirs = [f for f in file_infos if f.file_type == DIR] files = [f for f in file_infos if f.file_type == FILE] return sum([self.list_file(commit, d.file.path, recursive) for d in dirs], files) return list(file_infos) def glob_file(self, commit, pattern): """ ? :param commit: :param pattern: :return: A list of FileInfo objects """ r = self.stub.GlobFile(GlobFileRequest(commit=_commit_from(commit), pattern=pattern)) if hasattr(r, 'file_info'): return r.file_info return [] def delete_file(self, commit, path): """ deletes a file from a Commit. DeleteFile leaves a tombstone in the Commit, assuming the file isn't written to later attempting to get the file from the finished commit will result in not found error. The file will of course remain intact in the Commit's parent. :param commit: A tuple or string representing the commit :param path: The path to the file """ self.stub.DeleteFile(DeleteFileRequest(file=File(commit=_commit_from(commit), path=path))) def delete_all(self): self.stub.DeleteAll(google_dot_protobuf_dot_empty__pb2.Empty())
Ancestors (in MRO)
- PfsClient
- builtins.object
Static methods
def __init__(
self, host=None, port=None)
Creates a client to connect to Pfs
:param host: The pachd host. Default is 'localhost', which is used with pachctl port-forward
:param port: The port to connect to. Default is 30650
def __init__(self, host=None, port=None): """ Creates a client to connect to Pfs :param host: The pachd host. Default is 'localhost', which is used with `pachctl port-forward` :param port: The port to connect to. Default is 30650 """ # If a host or port is not specified, then try to set using environment variables or use the defaults. if host is None: host = os.environ.get('PACHD_SERVICE_HOST', 'localhost') if port is None: port = os.environ.get('PACHD_SERVICE_PORT_API_GRPC_PORT', '30650') self.channel = grpc.insecure_channel('{}:{}'.format(host, port)) self.stub = APIStub(self.channel)
def commit(
*args, **kwds)
A context manager for doing stuff inside a commit
@contextmanager def commit(self, repo_name, branch=None, parent=None): """ A context manager for doing stuff inside a commit """ commit = self.start_commit(repo_name, branch, parent) try: yield commit except Exception as e: print("An exception occurred during an open commit. " "Trying to finish it (Currently a commit can't be cancelled)") raise e finally: self.finish_commit(commit)
def create_repo(
self, repo_name, description=None)
Creates a new Repo object in pfs with the given name. Repos are the top level data object in pfs and should be used to store data of a similar type. For example rather than having a single Repo for an entire project you might have seperate Repos for logs, metrics, database dumps etc. :param repo_name: Name of the repo :param description: Repo description
def create_repo(self, repo_name, description=None): """ Creates a new Repo object in pfs with the given name. Repos are the top level data object in pfs and should be used to store data of a similar type. For example rather than having a single Repo for an entire project you might have seperate Repos for logs, metrics, database dumps etc. :param repo_name: Name of the repo :param description: Repo description """ self.stub.CreateRepo(CreateRepoRequest(repo=Repo(name=repo_name), description=description))
def delete_all(
self)
def delete_all(self): self.stub.DeleteAll(google_dot_protobuf_dot_empty__pb2.Empty())
def delete_branch(
self, repo_name, branch_name)
deletes a branch, but leaves the commits themselves intact. In other words, those commits can still be accessed via commit IDs and other branches they happen to be on. :param repo_name: The name of the repo :param branch_name: The name of the branch to delete
def delete_branch(self, repo_name, branch_name): """ deletes a branch, but leaves the commits themselves intact. In other words, those commits can still be accessed via commit IDs and other branches they happen to be on. :param repo_name: The name of the repo :param branch_name: The name of the branch to delete """ self.stub.DeleteBranch(DeleteBranchRequest(repo=Repo(name=repo_name), branch=branch_name))
def delete_commit(
self, commit)
deletes a commit.  Note it is currently not implemented. :param commit: A tuple or string representing the commit """ self.stub.DeleteCommit(DeleteCommitRequest(commit=_commit_from(commit)))
def delete_file(
self, commit, path)
deletes a file from a Commit. DeleteFile leaves a tombstone in the Commit, assuming the file isn't written to later attempting to get the file from the finished commit will result in not found error. The file will of course remain intact in the Commit's parent.
:param commit: A tuple or string representing the commit :param path: The path to the file
def delete_file(self, commit, path): """ deletes a file from a Commit. DeleteFile leaves a tombstone in the Commit, assuming the file isn't written to later attempting to get the file from the finished commit will result in not found error. The file will of course remain intact in the Commit's parent. :param commit: A tuple or string representing the commit :param path: The path to the file """ self.stub.DeleteFile(DeleteFileRequest(file=File(commit=_commit_from(commit), path=path)))
def delete_repo(
self, repo_name=None, force=False, all=False)
Deletes a repo and reclaims the storage space it was using. Note that as of 1.0 we do not reclaim the blocks that the Repo was referencing, this is because they may also be referenced by other Repos and deleting them would make those Repos inaccessible. This will be resolved in later versions. :param repo_name: The name of the repo :param force: if set to true, the repo will be removed regardless of errors. This argument should be used with care. :param all: Delete all repos
def delete_repo(self, repo_name=None, force=False, all=False): """ Deletes a repo and reclaims the storage space it was using. Note that as of 1.0 we do not reclaim the blocks that the Repo was referencing, this is because they may also be referenced by other Repos and deleting them would make those Repos inaccessible. This will be resolved in later versions. :param repo_name: The name of the repo :param force: if set to true, the repo will be removed regardless of errors. This argument should be used with care. :param all: Delete all repos """ if not all: if repo_name: self.stub.DeleteRepo(DeleteRepoRequest(repo=Repo(name=repo_name), force=force)) else: raise ValueError("Either a repo_name or all=True needs to be provided") else: if not repo_name: self.stub.DeleteRepo(DeleteRepoRequest(force=force, all=all)) else: raise ValueError("Cannot specify a repo_name if all=True")
def finish_commit(
self, commit)
Ends the process of committing data to a Repo and persists the Commit. Once a Commit is finished the data becomes immutable and future attempts to write to it with PutFile will error. :param commit: A tuple or string representing the commit
def finish_commit(self, commit): """ Ends the process of committing data to a Repo and persists the Commit. Once a Commit is finished the data becomes immutable and future attempts to write to it with PutFile will error. :param commit: A tuple or string representing the commit """ self.stub.FinishCommit(FinishCommitRequest(commit=_commit_from(commit)))
def flush_commit(
self, commits, repos=())
blocks until all of the commits which have a set of commits as provenance have finished. For commits to be considered they must have all of the specified commits as provenance. This in effect waits for all of the jobs that are triggered by a set of commits to complete. It returns an error if any of the commits it's waiting on are cancelled due to one of the jobs encountering an error during runtime. Note that it's never necessary to call FlushCommit to run jobs, they'll run no matter what, FlushCommit just allows you to wait for them to complete and see their output once they do. :param commits: A commit or a list of commits to wait on :param repos: Optional. Only the commits up to and including those repos will be considered, otherwise all repos are considered. :return: An iterator of CommitInfo objects
def flush_commit(self, commits, repos=tuple()): """ blocks until all of the commits which have a set of commits as provenance have finished. For commits to be considered they must have all of the specified commits as provenance. This in effect waits for all of the jobs that are triggered by a set of commits to complete. It returns an error if any of the commits it's waiting on are cancelled due to one of the jobs encountering an error during runtime. Note that it's never necessary to call FlushCommit to run jobs, they'll run no matter what, FlushCommit just allows you to wait for them to complete and see their output once they do. :param commits: A commit or a list of commits to wait on :param repos: Optional. Only the commits up to and including those repos will be considered, otherwise all repos are considered. :return: An iterator of CommitInfo objects """ return self.stub.FlushCommit(FlushCommitRequest(commit=[_commit_from(c) for c in commits], to_repo=[Repo(name=r) for r in repos]))
def get_file(
self, commit, path, offset_bytes=0, size_bytes=0, extract_value=True)
returns the contents of a file at a specific Commit.
:param commit: A tuple or string representing the commit :param path: The path of the file :param offset_bytes: Optional. specifies a number of bytes that should be skipped in the beginning of the file. :param size_bytes: Optional. limits the total amount of data returned, note you will get fewer bytes than size if you pass a value larger than the size of the file. If size is set to 0 then all of the data will be returned. :param extract_value: If True, then an ExtractValueIterator will be return, which will iterate over the bytes of the file. If False, then the Protobuf response iterator will return :return: An iterator over the file or an iterator over the protobuf responses
def get_file(self, commit, path, offset_bytes=0, size_bytes=0, extract_value=True): """ returns the contents of a file at a specific Commit. :param commit: A tuple or string representing the commit :param path: The path of the file :param offset_bytes: Optional. specifies a number of bytes that should be skipped in the beginning of the file. :param size_bytes: Optional. limits the total amount of data returned, note you will get fewer bytes than size if you pass a value larger than the size of the file. If size is set to 0 then all of the data will be returned. :param extract_value: If True, then an ExtractValueIterator will be return, which will iterate over the bytes of the file. If False, then the Protobuf response iterator will return :return: An iterator over the file or an iterator over the protobuf responses """ r = self.stub.GetFile(GetFileRequest(file=File(commit=_commit_from(commit), path=path), offset_bytes=offset_bytes, size_bytes=size_bytes)) if extract_value: return ExtractValueIterator(r) return r
def get_files(
self, commit, paths, recursive=False)
returns the contents of a list of files at a specific Commit.
:param commit: A tuple or string representing the commit :param paths: A list of paths to retrieve :param recursive: If True, will go into each directory in the list recursively :return: A dictionary of file paths and data
def get_files(self, commit, paths, recursive=False): """ returns the contents of a list of files at a specific Commit. :param commit: A tuple or string representing the commit :param paths: A list of paths to retrieve :param recursive: If True, will go into each directory in the list recursively :return: A dictionary of file paths and data """ filtered_file_infos = [] for path in paths: fi = self.inspect_file(commit, path) if fi.file_type == FILE: filtered_file_infos.append(fi) else: filtered_file_infos += self.list_file(commit, path, recursive=recursive) filtered_paths = [fi.file.path for fi in filtered_file_infos if fi.file_type == FILE] return {path: b''.join(self.get_file(commit, path)) for path in filtered_paths}
def glob_file(
self, commit, pattern)
? :param commit: :param pattern: :return: A list of FileInfo objects
def glob_file(self, commit, pattern): """ ? :param commit: :param pattern: :return: A list of FileInfo objects """ r = self.stub.GlobFile(GlobFileRequest(commit=_commit_from(commit), pattern=pattern)) if hasattr(r, 'file_info'): return r.file_info return []
def inspect_commit(
self, commit)
returns info about a specific Commit. :param commit: A tuple or string representing the commit :return: CommitInfo object
def inspect_commit(self, commit): """ returns info about a specific Commit. :param commit: A tuple or string representing the commit :return: CommitInfo object """ return self.stub.InspectCommit(InspectCommitRequest(commit=_commit_from(commit)))
def inspect_file(
self, commit, path)
returns info about a specific file. :param commit: A tuple or string representing the commit :param path: Path to file :return: A FileInfo object
def inspect_file(self, commit, path): """ returns info about a specific file. :param commit: A tuple or string representing the commit :param path: Path to file :return: A FileInfo object """ return self.stub.InspectFile(InspectFileRequest(file=File(commit=_commit_from(commit), path=path)))
def inspect_repo(
self, repo_name)
returns info about a specific Repo. :param repo_name: Name of the repo :return: A RepoInfo object
def inspect_repo(self, repo_name): """ returns info about a specific Repo. :param repo_name: Name of the repo :return: A RepoInfo object """ return self.stub.InspectRepo(InspectRepoRequest(repo=Repo(name=repo_name)))
def list_branch(
self, repo_name)
lists the active branches on a Repo :param repo_name: The name of the repo :return: A list of Branch objects
def list_branch(self, repo_name): """ lists the active branches on a Repo :param repo_name: The name of the repo :return: A list of Branch objects """ x = self.stub.ListBranch(ListBranchRequest(repo=Repo(name=repo_name))) if hasattr(x, 'branch_info'): return x.branch_info return []
def list_commit(
self, repo_name, to_commit=None, from_commit=None, number=0)
Lists commits.
:param repo_name: If only repo_name
is given, all commits in the repo are returned.
:param to_commit: optional. only the ancestors of to
, including to
itself,
are considered.
:param from_commit: optional. only the descendents of from
, including from
itself, are considered.
:param number: optional. determines how many commits are returned. If number
is 0,
all commits that match the aforementioned criteria are returned.
:return: A list of CommitInfo objects
def list_commit(self, repo_name, to_commit=None, from_commit=None, number=0): """ Lists commits. :param repo_name: If only `repo_name` is given, all commits in the repo are returned. :param to_commit: optional. only the ancestors of `to`, including `to` itself, are considered. :param from_commit: optional. only the descendents of `from`, including `from` itself, are considered. :param number: optional. determines how many commits are returned. If `number` is 0, all commits that match the aforementioned criteria are returned. :return: A list of CommitInfo objects """ req = ListCommitRequest(repo=Repo(name=repo_name), number=number) if to_commit is not None: req.to.CopyFrom(_commit_from(to_commit)) if from_commit is not None: getattr(req, 'from').CopyFrom(_commit_from(from_commit)) x = self.stub.ListCommit(req) if hasattr(x, 'commit_info'): return x.commit_info return []
def list_file(
self, commit, path, recursive=False)
Lists the files in a directory :param commit: A tuple or string representing the commit :param path: The path to the directory :param recursive: If True, continue listing the files for sub-directories :return: A list of FileInfo objects
def list_file(self, commit, path, recursive=False): """ Lists the files in a directory :param commit: A tuple or string representing the commit :param path: The path to the directory :param recursive: If True, continue listing the files for sub-directories :return: A list of FileInfo objects """ file_infos = self.stub.ListFile(ListFileRequest(file=File(commit=_commit_from(commit), path=path))).file_info if recursive: dirs = [f for f in file_infos if f.file_type == DIR] files = [f for f in file_infos if f.file_type == FILE] return sum([self.list_file(commit, d.file.path, recursive) for d in dirs], files) return list(file_infos)
def list_repo(
self)
Returns info about all Repos.
:param provenance: Optional. Specifies a set of provenance repos where only repos which have ALL of the specified repos as provenance will be returned. :return: A list of RepoInfo objects
def list_repo(self): """ Returns info about all Repos. :param provenance: Optional. Specifies a set of provenance repos where only repos which have ALL of the specified repos as provenance will be returned. :return: A list of RepoInfo objects """ x = self.stub.ListRepo(ListRepoRequest()) if hasattr(x, 'repo_info'): return x.repo_info return []
def provenances_for_repo(
self, repo_name)
def provenances_for_repo(self, repo_name): provenances = {} commits = self.list_commit(repo_name) sorted_commits = [x[0] for x in sorted([(c.commit.id, c.finished.seconds) for c in commits], key=lambda x: x[1])] for c in sorted_commits: for p in c.provenance: provenances[p.id] = c.commit.id return provenances
def put_file_bytes(
self, commit, path, value, delimiter=0, target_file_datums=0, target_file_bytes=0)
Uploads a binary bytes array as file(s) in a certain path
:param commit: A tuple or string representing the commit
:param path: Path in the repo the file(s) will be written to
:param value: The data bytes array, or an iterator returning chunked byte arrays
:param delimiter: Optional. causes data to be broken up into separate files with path
as a prefix.
:param target_file_datums: Optional. specifies the target number of datums in each written
file it may be lower if data does not split evenly, but will never be
higher, unless the value is 0.
:param target_file_bytes: specifies the target number of bytes in each written
file, files may have more or fewer bytes than the target.
def put_file_bytes(self, commit, path, value, delimiter=NONE, target_file_datums=0, target_file_bytes=0): """ Uploads a binary bytes array as file(s) in a certain path :param commit: A tuple or string representing the commit :param path: Path in the repo the file(s) will be written to :param value: The data bytes array, or an iterator returning chunked byte arrays :param delimiter: Optional. causes data to be broken up into separate files with `path` as a prefix. :param target_file_datums: Optional. specifies the target number of datums in each written file it may be lower if data does not split evenly, but will never be higher, unless the value is 0. :param target_file_bytes: specifies the target number of bytes in each written file, files may have more or fewer bytes than the target. """ if _is_iterator(value): def _wrap(v): for x in v: yield PutFileRequest(file=File(commit=_commit_from(commit), path=path), value=x, delimiter=delimiter, target_file_datums=target_file_datums, target_file_bytes=target_file_bytes) self.stub.PutFile(_wrap(value)) return def _blocks(v): for i in range(0, len(v), BUFFER_SIZE): yield PutFileRequest(file=File(commit=_commit_from(commit), path=path), value=v[i:i + BUFFER_SIZE], delimiter=delimiter, target_file_datums=target_file_datums, target_file_bytes=target_file_bytes) self.stub.PutFile(_blocks(value))
def put_file_url(
self, commit, path, url, recursive=False)
puts a file using the content found at a URL. The URL is sent to the server which performs the request.
:param commit: A tuple or string representing the commit :param path: The path to the file :param url: The url to download :param recursive: allow for recursive scraping of some types URLs for example on s3:// urls.
def put_file_url(self, commit, path, url, recursive=False): """ puts a file using the content found at a URL. The URL is sent to the server which performs the request. :param commit: A tuple or string representing the commit :param path: The path to the file :param url: The url to download :param recursive: allow for recursive scraping of some types URLs for example on s3:// urls. """ self.stub.PutFile(iter([PutFileRequest(file=File(commit=_commit_from(commit), path=path), url=url, recursive=recursive)]))
def set_branch(
self, commit, branch_name)
sets a commit and its ancestors as a branch :param commit: A tuple or string representing the commit :param branch_name: The name for the branch to set
def set_branch(self, commit, branch_name): """ sets a commit and its ancestors as a branch :param commit: A tuple or string representing the commit :param branch_name: The name for the branch to set """ self.stub.SetBranch(SetBranchRequest(commit=_commit_from(commit), branch=branch_name))
def start_commit(
self, repo_name, branch=None, parent=None)
Begins the process of committing data to a Repo. Once started you can write to the Commit with PutFile and when all the data has been written you must finish the Commit with FinishCommit. NOTE, data is not persisted until FinishCommit is called. :param repo_name: The name of the repo :param branch: is a more convenient way to build linear chains of commits. When a commit is started with a non empty branch the value of branch becomes an alias for the created Commit. This enables a more intuitive access pattern. When the commit is started on a branch the previous head of the branch is used as the parent of the commit. :param parent: specifies the parent Commit, upon creation the new Commit will appear identical to the parent Commit, data can safely be added to the new commit without affecting the contents of the parent Commit. You may pass "" as parentCommit in which case the new Commit will have no parent and will initially appear empty. :return: Commit object
def start_commit(self, repo_name, branch=None, parent=None): """ Begins the process of committing data to a Repo. Once started you can write to the Commit with PutFile and when all the data has been written you must finish the Commit with FinishCommit. NOTE, data is not persisted until FinishCommit is called. :param repo_name: The name of the repo :param branch: is a more convenient way to build linear chains of commits. When a commit is started with a non empty branch the value of branch becomes an alias for the created Commit. This enables a more intuitive access pattern. When the commit is started on a branch the previous head of the branch is used as the parent of the commit. :param parent: specifies the parent Commit, upon creation the new Commit will appear identical to the parent Commit, data can safely be added to the new commit without affecting the contents of the parent Commit. You may pass "" as parentCommit in which case the new Commit will have no parent and will initially appear empty. :return: Commit object """ return self.stub.StartCommit(StartCommitRequest(parent=Commit(repo=Repo(name=repo_name), id=parent), branch=branch))
def subscribe_commit(
self, repo_name, branch, from_commit_id=None)
SubscribeCommit is like ListCommit but it keeps listening for commits as they come in.
:param repo_name: Name of the repo :param branch: Branch to subscribe to :param from_commit_id: Optional. only commits created since this commit are returned
:return: Iterator of Commit objects
def subscribe_commit(self, repo_name, branch, from_commit_id=None): """ SubscribeCommit is like ListCommit but it keeps listening for commits as they come in. :param repo_name: Name of the repo :param branch: Branch to subscribe to :param from_commit_id: Optional. only commits created since this commit are returned :return: Iterator of Commit objects """ repo = Repo(name=repo_name) req = SubscribeCommitRequest(repo=repo, branch=branch) if from_commit_id is not None: getattr(req, 'from').CopyFrom(Commit(repo=repo, id=from_commit_id)) return self.stub.SubscribeCommit(req)
Instance variables
var channel
var stub
class PutFileRecord
A ProtocolMessage
Ancestors (in MRO)
- PutFileRecord
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var OBJECT_HASH_FIELD_NUMBER
var OVERWRITE_INDEX_FIELD_NUMBER
var SIZE_BYTES_FIELD_NUMBER
class PutFileRecords
A ProtocolMessage
Ancestors (in MRO)
- PutFileRecords
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var FOOTER_FIELD_NUMBER
var HEADER_FIELD_NUMBER
var RECORDS_FIELD_NUMBER
var SPLIT_FIELD_NUMBER
var TOMBSTONE_FIELD_NUMBER
class PutFileRequest
A ProtocolMessage
Ancestors (in MRO)
- PutFileRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DELIMITER_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var FILE_FIELD_NUMBER
var HEADER_RECORDS_FIELD_NUMBER
var OVERWRITE_INDEX_FIELD_NUMBER
var RECURSIVE_FIELD_NUMBER
var TARGET_FILE_BYTES_FIELD_NUMBER
var TARGET_FILE_DATUMS_FIELD_NUMBER
var URL_FIELD_NUMBER
var VALUE_FIELD_NUMBER
class PutObjectRequest
A ProtocolMessage
Ancestors (in MRO)
- PutObjectRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BLOCK_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var TAGS_FIELD_NUMBER
var VALUE_FIELD_NUMBER
class Repo
A ProtocolMessage
Ancestors (in MRO)
- Repo
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var NAME_FIELD_NUMBER
class RepoAuthInfo
A ProtocolMessage
Ancestors (in MRO)
- RepoAuthInfo
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var ACCESS_LEVEL_FIELD_NUMBER
var DESCRIPTOR
var Extensions
class RepoInfo
A ProtocolMessage
Ancestors (in MRO)
- RepoInfo
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var AUTH_INFO_FIELD_NUMBER
var BRANCHES_FIELD_NUMBER
var CREATED_FIELD_NUMBER
var DESCRIPTION_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var REPO_FIELD_NUMBER
var SIZE_BYTES_FIELD_NUMBER
class StartCommitRequest
A ProtocolMessage
Ancestors (in MRO)
- StartCommitRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BRANCH_FIELD_NUMBER
var DESCRIPTION_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var PARENT_FIELD_NUMBER
var PROVENANCE_FIELD_NUMBER
class SubscribeCommitRequest
A ProtocolMessage
Ancestors (in MRO)
- SubscribeCommitRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var BRANCH_FIELD_NUMBER
var DESCRIPTOR
var Extensions
var FROM_FIELD_NUMBER
var REPO_FIELD_NUMBER
var STATE_FIELD_NUMBER
class Tag
A ProtocolMessage
Ancestors (in MRO)
- Tag
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var NAME_FIELD_NUMBER
class TagObjectRequest
A ProtocolMessage
Ancestors (in MRO)
- TagObjectRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var OBJECT_FIELD_NUMBER
var TAGS_FIELD_NUMBER
class WalkFileRequest
A ProtocolMessage
Ancestors (in MRO)
- WalkFileRequest
- google.protobuf.pyext._message.CMessage
- google.protobuf.message.Message
- builtins.object
Class variables
var DESCRIPTOR
var Extensions
var FILE_FIELD_NUMBER


Monte Carlo Methods
-
Tag:
- Overview
- Draw random samples uniformly over the cumulative distribution function (CDF)
- Markov Chain Monte-Carlo (MCMC)
- Reference
Overview
Draw random samples uniformly over the cumulative distribution function (CDF)
Markov Chain Monte-Carlo (MCMC)
Markov chain Monte-Carlo uses a homogeneous, ergodic and reversible (see [1] for the meaning of the adjectives) Markov chain to generate consistent samples draw from any given distribution.
Glossary
-
Stationary. A markov chain is stationary if does not depend on .
-
Ergodic. A markov chain is ergodic if converges to a distribution not matter what the starting value is.
-
Irreducible. A markov chain is irreducible if the samples of the chain can move all over the state space, i.e., can eventually reach any region of the state space, no matter its initial value.
-
Aperiodic. Sampling不会周期性地回到某个点.
Metropolis–Hastings algorithm
-
Given , generate , where the conditional density is called proposal, which is used to generate next sample.
-
Take with probability or with probability , where . Here we do not need to obtain the exact value of and , but more easily, their ratio.
(The proposal must be carefully chosen to ensure that the samples of MCMC converge to the target distribution .)
Gibbs sampling
Gibbs sampling is a multivariate version of Metropolis–Hastings algorithm. Here we consider the multivariate distribution of .
In the transition step （step 2 above）, we generate next proposal from based on the conditional density , since the variables are correlated. It means the th component is conditioned on all the other components at hand.
Metropolis within Gibbs (Hybrid Gibbs)
Metropolis within Gibbs uses the Metropolis Hasting proposal for each component of .
http://www.chadfulton.com/fulton_statsmodels_2017/sections/5-posterior_simulation.html
Hamiltonian Monte Carlo (Hybrid Monte Carlo)
Hamiltonian dynamics是一种物理学上的动态过程,描述了一个有质量的物体的势能和动能的相互转化.想像一个固定在弹簧末端的小球,在小球左右来回摆动时,小球的动能和弹簧的弹性势能相互转化.其中弹性势能的计算取决于小球的位置,动能的计算取决于小球的动量,这两个值定义了Hamiltonian dynamics动态过程的一个坐标.随着时间的变化,每一个时间点的坐标可以通过积分的方法计算得到.例如,Leap Frog Method就是一个将连续时间离散为很小的时间段计算每个离散时间点的方法. 对于一个分布来讲,不同的位置的总能量(势能+动能)是不同的,能量的大小反映了概率分布.对于MAP问题,就是要找能量最低的那个点.
在Hamiltonian Monte Carlo(HMC)采样中,Hamiltonian dynamics被用来生成下一个proposal - .此时的energy funciton定义为势能和动能之和,即.采样点的概率为.整个proposal的过程如下:
- 在第步,给定一个采样,从变量的标准分布(canonical distribution,人为定义)中随机采样一个值构成坐标.
- 跑步的Leap Frog Method来模拟Hamiltonian dynamics动态过程,到达结束位置
- 比较和的相对大小,按照Metropolis–Hastings algorithm的方法accept或者reject,从而得到.
在每一步生成proposal时,因为引入了随机动量,所以HMC可以有效地探索整个分布空间. 因为HMC相邻两个采样的距离比Metropolis–Hastings algorithm更大,且相关性(autocorrelation)更小,所以有更快的收敛效果.
Adaptive rejection sampling
The main idea is to asymptotically optimize the transition function to minimize the rejection rate.
Cluster sampling
It split an image into individual clusters, which is useful for image segmentation.
Interpretations of Cluster Sampling by Swendsen-Wang
Random Walk
Google Page Rank via random walk
Convergence
The structural properties required for a constructed chain to converge appropriately are well understood: essentially irreducibility and aperiodicity.
Reference
[1] Lecture notes
[2] Introduction to Markov Chain Monte Carlo
[3] Very useful for the newers: A simple introduction to Markov Chain Monte–Carlo sampling


Contents:
stream2py
stream2py.buffer_reader
stream2py.protocols
stream2py.simply
stream2py.source_reader
stream2py.sources
stream2py.sources.http
stream2py.stream_buffer
stream2py.stream_source
stream2py.tests.test_buffer_reader
stream2py.tests.test_lock_sorted_deque
stream2py.tests.test_source_reader
stream2py.tests.test_stream_buffer
stream2py.tests.test_stream_source
stream2py.tests.test_util
stream2py.tests.utils_for_testing
stream2py.util
stream2py.utility.locked_sorted_deque
stream2py.utility.reader_writer_lock
stream2py.utility.sorted_collection
stream2py.utility.sorted_deque
stream2py.utility.typing_hints


DeepGIFs: Using Deep Learning to Understand and Synthesize Motion
Humans are adept at intuiting how objects can and should move. We form this understanding before we form
our understanding of the physical mechanisms underpinning motion. Understanding `rules of motion' is central to the task
of generating realistic sequences. Frames should plausibly follow from previous frames, and objects should move, deform,
and interact realistically. Structuring a video generation task around consistent motion is a challenging task for which
deep learning is a well-suited solution and for our CS 231N final project at Stanford,
we did exactly that.
Abstract
Recurrent neural networks (RNNs) have seen widespread use across a variety of generative tasks requiring temporality. In this paper we explore the effectiveness of deep recurrent networks in the task of understanding
motion. In particular, we propose a modular architecture for inferring realistic motion from still `seed images', based on
the recent success of generative adversarial networks. We evaluate this architecture on a variety of synthetic datasets and
show that our network is able to successfully learn motion and generate new videos from still images.
Architecture
Our model consists of a convolutional encoder and decoder network, an LSTM, and a discriminator network. Our architecture
uses four conv-relu-pool layers to encode the seed frame into a latent representation on which the LSTM predicts future frames.
The decoder network converts these predictions back into images and has the inverse architecture of the encoder. The discriminator
has the same architecture as the encoder with a sigmoid function applied to the final scalar output. It is used to discriminate between
the generated frames and the original data. This allows our network to generate clearer outputs.
Results: Bouncing Ball
Each pair of images consists of an actual GIF in our dataset on the left and a generated GIF
on the right, produced by taking the first frame of the actual GIF and inputting it to our model.
Results: Moving MNIST
Each pair of images consists of an actual GIF in our dataset on the left and a generated GIF on the right, produced by taking
the first frame of the actual GIF and inputting it to our model.


Fee Structure
Current Fee Structure
Scheduled Fee Structure
Online / Phone
First Time(Rs.)
Follow Up(Rs.)
Extension(Rs.)
In Clinic
First Time(Rs.)
Follow Up(Rs.)
Effective Date :
Allow patient to pay at clinic


PlutoX
PlutoX aims to be the CMS solution for
the next generation of Web sites.
We will continue to develop the CMS solution in the future.
PlutoX
PlutoX aims to be the CMS solution for
the next generation of Web sites.
We will continue to develop the CMS solution in the future.
PlutoX
PlutoX aims to be the CMS solution for
the next generation of Web sites.
We will continue to develop the CMS solution in the future.


{{ define "title" }} {{ .Title }} – {{ .Site.Title }} {{ end }} {{ define "main" }}
F5 was having difficulty engaging and reaching the global security audience.
hypothesis and solutionF5 was having difficulty engaging and reaching the global security audience.
key responsibilitiesF5 was having difficulty engaging and reaching the global security audience.
F5 was having difficulty engaging and reaching the global security audience.


As the subtitle suggests, this script brings the power of events from platforms such as node.js to your browser. Although it can be used on any other platform, I just built it with browsers in mind.
This is actually the fourth full rewrite of EventEmitter, my aim is for it to be faster and lighter than ever before. It also has a remapped API which just makes a lot more sense. Because the methods now have more descriptive names it is friendlier to extension into other classes. You will be able to distinguish event method from your own methods.
I have been working on it for over ~~a year~~ two years so far and in that time my skills in JavaScript have come a long way. This script is a culmination of my learnings which you can hopefully find very useful.
There are no hard dependencies. The only reason you will want to run npm install
to grab the development dependencies is to build the documentation or minify the source code. No other scripts are required to actually use EventEmitter.
You can run tools/doc.sh
to build from the JSDoc comments found within the source code. The built documentation will be placed in docs/api.md
. I actually keep this inside the repository so each version will have it's documentation stored with it.
You can grab minified versions of EventEmitter from inside this repository, every version is tagged. If you need to build a custom version then you can run tools/dist.sh
.
You can clone the repository with your generic clone commands as a standalone repository or submodule.
```bash # Full repository git clone git://github.com/Wolfy87/EventEmitter.git
git submodule add git://github.com/Wolfy87/EventEmitter.git assets/js/EventEmitter ```
You can also get a copy of EventEmitter through the following package managers: * NPM (wolfy87-eventemitter) * Bower (eventEmitter) * Component (Wolfy87/EventEmitter)
Tests are performed using Mocha and Chai in the following browsers.
When testing in the more modern browsers, I run it through the very early versions, some midrange versions and the very latest ones too. I don't just do the latest version.
EventEmitter will always be tested and working perfectly in all of them before a release. I will not release anything I think is riddled with bugs. However, if you do spot one, please submit it as an issue and I will get right on it.
I had to stop testing in IE<9. This is because Jasmine no longer seems to work in old IE. I converted all of my tests to Mocha/Chai and then realised Chai doesn't work in IE<9 either. I know it works, I just can't run the tests anymore, I still recommend using it in older IE versions. However, if you do encounter problems, all you have to do is roll back before v4.1.0, the first IE9+ only tested version.
As consolation for the lack of legacy browser testing, I have made sure the examples listed above run perfectly in IE6+. So the unit tests might not be able to run, but I am very confident that it works absolutely perfectly.
If you wish to contribute to the project then please commit your changes into the develop
branch. All pull requests should contain a failing test which is then resolved by your additions. A perfect example was submitted by nathggns.
Copyright (c) 2011-2013 Oliver Caldwell
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
 IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


JavaScript is disabled on your browser.
Package
Class
Use
Tree
Deprecated
Index
Help
Prev Letter
Next Letter
Frames
No Frames
All Classes
A
B
C
D
E
F
G
H
I
J
L
M
N
P
R
S
T
U
V
W
Y
P
Paper
- Class in
<Unnamed>
Paper()
- Constructor for class
Paper
passwordField
- Variable in class
BankAccountTestUI
payment
- Variable in class
CashRegister
PhoneNumberFormat
- Class in
<Unnamed>
PhoneNumberFormat()
- Constructor for class
PhoneNumberFormat
pinNumber
- Variable in class
BankAccount
Stores an integer PIN number for a BankAccount.
pointA
- Variable in class
Triangle
pointB
- Variable in class
Triangle
pointC
- Variable in class
Triangle
posX
- Static variable in class
Critter
posY
- Static variable in class
Critter
price
- Variable in class
Product
printLogMessages
- Static variable in class
Critter
printReceipt()
- Method in class
CashRegister
printResults(Critter)
- Static method in class
CritterTester
Product
- Class in
<Unnamed>
Product.java - represents a product.
Product(String, double)
- Constructor for class
Product
productName
- Variable in class
Product
ProductPrinter
- Class in
<Unnamed>
ProductPrinter.java - interfaces with
Product
ProductPrinter()
- Constructor for class
ProductPrinter
prompt
- Static variable in class
Architect
prompt
- Static variable in class
AreaTester
prompt
- Static variable in class
BankAccountTester
Scanner for console interface to BankAccount class.
prompt
- Static variable in class
BasicCarAnalyst
prompt
- Static variable in class
Bookstore
prompt
- Static variable in class
CarAnalyst
Primary input for CarAnalyst; may change as UI is developed.
prompt
- Static variable in class
ChangeCalculator
prompt
- Static variable in class
DewTempCalculator
The Scanner to read all values for the program
prompt
- Static variable in class
DieSimulator
prompt
- Static variable in class
DigitCounter
Scanner to read all user inputs
prompt
- Static variable in class
EasterCalculator
prompt
- Static variable in class
Greeter
prompt
- Static variable in class
JCowsay
prompt
- Static variable in class
LargeNumberFormatter
prompt
- Static variable in class
LargeNumberFormatterLong
prompt
- Static variable in class
MeterConverter
prompt
- Static variable in class
Namebox
prompt
- Static variable in class
PhoneNumberFormat
prompt
- Static variable in class
ReverseTester
prompt
- Static variable in class
Statistics
prompt
- Static variable in class
Thermistor
prompt
- Static variable in class
TriangleCreator
properties
- Variable in class
CarAnalyst
Used to list the following user-given properties for the car: Cost of car, fuel efficiency, gas price, mileage after five years, and estimated value after five years.
purchase
- Variable in class
CashRegister
A
B
C
D
E
F
G
H
I
J
L
M
N
P
R
S
T
U
V
W
Y
Package
Class
Use
Tree
Deprecated
Index
Help
Prev Letter
Next Letter
Frames
No Frames
All Classes


Three.js Examples
The goal of this collection is to provide a set of basic and instructive examples that introduce the various features in


GHBC wants to make a lasting difference in your life, in our community, and in the world.
Our mission is to lead people to become fully devoted followers of Christ. That's how we're able to make a difference, and it's the driving force behind everything we do.
A church isn't a building—it's the people. We meet in locations around the nation and globally online. No matter where you join us, we think you'll find friendly people who are excited to get to know you!
 We want to help you become the person God has created you to be. Your journey has a next step and we'll help you find it. It could start with serving, finding God's vision for your life, giving, or joining us regularly.
You don't have to take the next step alone. With a strong community of friends, you can laugh, grow, and serve with people who genuinely care about you. Find a small group, or LifeGroup, to join you on your journey.
Your Family
Strong families start with solid foundations. That's why we partner with parents to help raise the next generation to become followers of Christ. Your children and teens have opportunities throughout the week to grow relationships with each other and with God.
Our community and world
We are passionate about serving the global community and demonstrating God's love to others. Whether in person or online, individually or in groups, within our church and around the world, we are focused on supporting and engaging in relationships that provide relief and restoration to a hurting world.
Our leadership
Our Directional Leadership Team works together to shape the vision and direction of GHBC.
Apostle Elias Emmanuel
Senior Pastor
As the Senior Pastor of GHBC, Elias provides teaching, leadership, and guidance for the church as a whole. Following a vision God gave him for a different kind of church, he and a handful of people launched GHBC in 1992.
He speaks frequently at conferences, has written several books including his latest release, Prophetic .... He and his wife, Evang.(Mrs) Mary Elias, live in the calabar, Cross River State and have two children.
Pastor Samuel Bassey
Church Admin
He is responsible for the administration of the church. Sam and his wife, Abigail, live in Calabar, CRS and have four children.
Pastor Ewa Henshaw
Resident Pastor
He is passionate about exploring new ideas and finding practical ways to leverage them for the global Church. He has helped pioneer the missional and evangelical move of the church.
Min. Udofia
Choir Director
She is the choir director and ...... She and her husband, Pastor Udofia, live in Calabar, CRS with their four children.
Our Beliefs


| Home | Trees | Indices | Help |
|
|---|
|
|
object --+ | named._Named --+ | named._NamedBase --+ | gxbases._GeodesicBase --+ | _GeodesicLineExact
(INTERNAL) Base class for GeodesicLineExact.
| Instance Methods | |||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
Inherited from Inherited from Inherited from |
| Class Variables | |
|
Inherited from |
| Properties | |
|
a13 Get (spherical) arc length from the first to the reference point ( degrees ).
|
|
|
azi0 Get the equatorial azimuth, the azimuth of this geodesic line as it crosses the equator in a northward direction ( degrees90 ).
|
|
|
azi0_sincos2 Get the sine and cosine of the equatorial azimuth (2-tuple (sin, cos) ).
|
|
|
azi1 Get the azimuth at the first point (compass degrees ).
|
|
|
azi1_sincos2 Get the sine and cosine of the first point's azimuth (2-tuple (sin, cos) ).
|
|
|
caps Get the capabilities (bit-or'ed Caps ).
|
|
|
geodesic Get the exact geodesic (GeodesicExact). |
|
|
lat1 Get the latitude of the first point ( degrees ).
|
|
|
lon1 Get the longitude of the first point ( degrees ).
|
|
|
s13 Get the distance from the first to the reference point C({meter}). |
|
|
Inherited from Inherited from Inherited from |
| Method Details |
(INTERNAL) New
|
Find the position on the line given
Note:
By default, |
Check the available capabilities.
|
Find the position on the line given
Notes:
|
Set reference point 3 in terms of distance to the first point.
|
Set reference point 3 in terms of distance to the first point.
|
Return this
|
| Property Details |
a13Get (spherical) arc length from the first to the reference point
(
See Also: Method SetArc. |
azi0Get the equatorial azimuth, the azimuth of this geodesic line
as it crosses the equator in a northward direction
(
|
azi0_sincos2Get the sine and cosine of the equatorial azimuth (2-tuple
|
azi1Get the azimuth at the first point (compass
|
azi1_sincos2Get the sine and cosine of the first point's azimuth (2-tuple
|
capsGet the capabilities (bit-or'ed
|
geodesicGet the exact geodesic (GeodesicExact).
|
lat1Get the latitude of the first point (
|
lon1Get the longitude of the first point (
|
s13Get the distance from the first to the reference point C({meter}).
See Also: Method SetDistance. |
| Home | Trees | Indices | Help |
|
|---|
| Generated by Epydoc 3.0.1 on Thu Sep 30 12:33:46 2021 | http://epydoc.sourceforge.net |


Pregnancy tracking app
Built the first prototype for a science-based pregnancy tracking app with Flutter and Firebase.
Cycle tracking app
Cycle tracking apps are used by lots of people, yet they're often not designed for them but rather at them. drip is different in that it leaves users' data on their device and ecourages them to think for themselves rather than trust an app's algorithm.
A secure Scuttlebutt client for building your own decentralized social network
Scuttlebutt is a decentralized social network that functions completely peer-to-peer. It is possible to create lots of different networks for different groups and purposes, but that process requires a bit of technical skill - blockparty addresses this by making it easy to set up a new network and invite people to it. It's also a cool client for any Scuttlebutt network.
I've worked as a Javascript backend developer and coding teacher and I really like getting to the bottom of things and passing that knowledge onto others. "Aha!" moments are my jam - whether for myself or for others.


At the tender age of eight, my mother took me to see "The Amityville Horror" in a movie theatre. I remember being exceptionally scared by it, and recall quite vividly some of the movie's scary and eerie moments like the black liquid oozing from a toilet; the pig with the red eyes; the Red Room scene, and so on. I had not seen the film since 1979 until recently on DVD. Well, looking at the film now more objectively than when I was eight, I have concluded that "The Amityville Horror" is only occasionally effective but it also mostly silly, dull, and frightfully cold and emotionless.
The story, based on a now debunked "true story" novel by Jay Anson, centers on the Lutzes, a couple who move in to the dreaded Long Island house with their three kids. The house is sold cheaply, mainly because a killing spree took place a few years earlier involving the DeFeo family - they were killed by their teenage son with a shotgun. I would never move into a house where murders had taken place - nowadays, such houses are usually boarded up though not always. Anyway, the Lutzes love the house and buy it. George Lutz (James Brolin) begins to feel the curious need of chopping wood for the fireplace through most of the film. The wife feels neglected, especially since he won't help with carrying the groceries. The daughter starts talking to her new friend, Jody, who is invisible, and has a habit of locking her babysitter in the closet. A frantic psychotherapist/priest/exorcist (Rod Steiger) comes to bless the house one day until he is chased out by flies and a voice saying, "Get Out!"  Either the house smells and needs to be fumigated or it is haunted.
"The Amityville Horror" is chock full of "shock" moments, and some of them are effective (the babysitter pounding on the closet door with her bloodied hands is disturbing). The film, however, has no momentum. It sort of labors along at a snail pace pausing occasionally for some weird occurrences and rappings in the house (my favorite being the case of the missing 1500 dollars). The main flaw is that the Lutzes are presented as less a family than some annoying neighbors I could hardly care about - no attempt to develop sympathy for them has been made. Mr. Brolin comes off as demented a madman within the first few minutes as Jack Nicholson did in "The Shining." Margot Kidder, a reliable actress of dignity, comes off best but not enough to save the film. Steiger overacts shamefully, supposedly trying to match the towering performance of Max Von Sydow in "The Exorcist." As the realtor should have said at the beginning of the film: No sale.
P.S. The Oscar-nominated musical score by Lalo Schifrin was originally composed for "The Exorcist" but it was rejected by director William Friedkin.
For more reviews, check out JERRY AT THE MOVIES at http://buffs.moviething.com/buffs/faust/
E-mail me with any questions, comments or general complaints at<EMAIL_ADDRESS>or at<EMAIL_ADDRESS>The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


2 Thessalonians 1:5
Which is a manifest token of the righteous judgment of God, that ye may be counted worthy of the kingdom of God, for which ye also suffer:


Love can be a powerful thing. It fills the time around your wedding with kindness and generosity. People spend hundreds on engagement parties, stag/hen nights, bridal showers and gift registries. So, why not use the power of love to make a difference to those who need it most?
SoGive is an online giving platform with a difference. We’ve built the largest database of charity impact data, to help you find the right charity to support and to show you, your family and friends the real impact your gifts have.
We’re 100% free – for you, and for the charities (apart from credit-card fees, there's nothing we can do about those). Other platforms take 5% or have hidden fees – we only suggest donors leave a tip.