sevenbridges
is an R/Bioconductor package that provides an interface for Seven Bridges public API. The supported platforms includes the Seven Bridges Platform, Cancer Genomics Cloud (CGC), and Cavatica.
Learn more from our documentation on the Seven Bridges Platform and the Cancer Genomics Cloud (CGC).
The sevenbridges
package only supports v2+ versions of the API, since versions prior to V2 are not compatible with the Common Workflow Language (CWL). This package provides a simple interface for accessing and trying out various methods.
There are two ways of constructing API calls. For instance, you can use low-level API calls which use arguments like path
, query
, and body
. These are documented in the API reference libraries for the Seven Bridges Platform and the CGC. An example of a low-level request to “list all projects” is shown below. In this request, you can also pass query
and body
as a list.
library("sevenbridges") a <- Auth(token = "your_token", platform = "aws-us") a$api(path = "projects", method = "GET")
(Advanced user option) The second way of constructing an API request is to directly use the httr
package to make your API calls, as shown below.
a$project()
Before we start, keep in mind the following:
offset
and limit
Every API call accepts two arguments named offset
and limit
.
By default, offset
is set to 0
and limit
is set to 100
. As such, your API request returns the first 100 items when you list items or search for items by name. To search and list all items, use complete = TRUE
in your API request.
Search by ID
When searching by ID, your request will return your exact resource as it is unique. As such, you do not have to set offset
and limit
manually. It is a good practice to find your resources by their ID and pass this ID as an input to your task. You can find a resource’s ID in the final part of the URL on the visual interface or via the API requests to list resources or get a resource’s details.
Search by name
Search by name returns all partial matches unless you specify exact = TRUE
.
The sevenbridges
package is available on both the release
and devel
branch from Bioconductor.
To install it from the release
branch, use:
install.packages("BiocManager") BiocManager::install("sevenbridges")
To install it from the devel
branch, use:
install.packages("BiocManager") BiocManager::install("sevenbridges", version = "devel")
Since we are constantly improving our API and client libraries, please also visit our GitHub repository for the most recent news and the latest version of the package.
If you do not have devtools
This installation requires that you have the devtools
package. If you do not have this package, you can install it from CRAN.
install.packages("devtools")
You may get an error for missing system dependencies such as curl
and openssl
. For example, in Ubuntu, you probably need to do the following first to install devtools
and to build vignettes since you need pandoc
.
apt-get update
apt-get install libcurl4-gnutls-dev libssl-dev pandoc pandoc-citeproc
If devtools
is already installed
Install the latest version for sevenbridges
from GitHub with the following:
install.packages("BiocManager") install.packages("readr") devtools::install_github( "sbg/sevenbridges-r", repos = BiocManager::repositories(), build_vignettes = TRUE, dependencies = TRUE )
If you have trouble with pandoc
and do not want to install it, set build_vignettes = FALSE
to avoid the vignettes build.
For more details about how to use the API client in R, please consult the Seven Bridges API Reference section below for a complete guide.
Auth
ObjectBefore you can access your account via the API, you have to provide your credentials. You can obtain your credentials in the form of an “authentication token” from the Developer Tab under Account Settings on the visual interface. Once you’ve obtained this, create an Auth
object, so it remembers your authentication token and the path for the API. All subsequent requests will draw upon these two pieces of information.
Let’s load the package first:
You have three different ways to provide your token. Choose from one method below:
Auth()
.SB_API_ENDPOINT
and SB_AUTH_TOKEN
.$HOME/.sevenbridges/credentials
, provides an organized way to collect and manage all your API authentication information for Seven Bridges platforms.Method 1: Direct authentication
This is the most common method to construct the Auth
object. For example:
(a <- Auth(platform = "cgc", token = "your_token"))
Using platform: cgc
== Auth ==
url : https://cgc-api.sbgenomics.com/v2/
token : <your_token>
Method 2: Environment variables
To set the two environment variables in your system, you could use the function sbg_set_env()
. For example:
sbg_set_env("https://cgc-api.sbgenomics.com/v2", "your_token")
Note that this change might be just temporary, please feel free to use the standard method to set persistent environment variables according to your operating system.
Create an Auth
object:
a <- Auth(from = "env")
Method 3: User configuration file
Assume we have already created the configuration file named credentials
under the directory $HOME/.sevenbridges/
:
[aws-us-rfranklin]
api_endpoint = https://api.sbgenomics.com/v2
auth_token = token_for_this_user
# This is a comment:
# another user on the same platform
[aws-us-rosalind-franklin]
api_endpoint = https://api.sbgenomics.com/v2
auth_token = token_for_this_user
[default]
api_endpoint = https://cgc-api.sbgenomics.com/v2
auth_token = token_for_this_user
[gcp]
api_endpoint = https://gcp-api.sbgenomics.com/v2
auth_token = token_for_this_user
To load the user profile aws-us-rfranklin
from this configuration file, simply use:
a <- Auth(from = "file", profile_name = "aws-us-rfranklin")
If profile_name
is not specified, we will try to load the profile named [default]
:
a <- Auth(from = "file")
Note: API paths (base URLs) differ for each Seven Bridges environment. Be sure to provide the correct path for the environment you are using. API paths for some of the environments are:
| Platform Name | API Base URL | Short Name |
|---|---|---|
| Seven Bridges Platform (US) | https://api.sbgenomics.com/v2 |
"aws-us" |
| Seven Bridges Platform (EU) | https://eu-api.sbgenomics.com/v2 |
"aws-eu" |
| Seven Bridges Platform (China) | https://api.sevenbridges.cn/v2 |
"ali-cn" |
| Cancer Genomics Cloud (CGC) | https://cgc-api.sbgenomics.com/v2 |
"cgc" |
| Cavatica | https://cavatica-api.sbgenomics.com/v2 |
"cavatica" |
| BioData Catalyst Powered by Seven Bridges | https://api.sb.biodatacatalyst.nhlbi.nih.gov/v2 |
"f4c" |
Please refer to the API reference section for more usage and technical details about the three authentication methods.
This call returns information about your account.
a$user()
== User ==
href : https://cgc-api.sbgenomics.com/v2/users/RFranklin
username : RFranklin
email<EMAIL_ADDRESS>first_name : Rosalind
last_name : Franklin
affiliation : Seven Bridges Genomics
country : United States
Get information about a user
This call returns information about the specified user. Note that currently you can view only your own user information, so this call is equivalent to the call to get information about your account.
a$user(username = "RFranklin")
This call returns information about your current rate limit. This is the number of API calls you can make in one hour.
a$rate_limit()
== Rate Limit ==
limit : 1000
remaining : 993
reset :<PHONE_NUMBER>
Each project must have a Billing Group associated with it. This Billing Group pays for the storage and computation in the project.
For example, your first project(s) were created with the free funds from the Pilot Funds Billing Group assigned to each user at sign-up. To get information about billing:
# check your billing info a$billing() a$invoice()
For more information, use breakdown = TRUE
.
a$billing(id = "your_billing_id", breakdown = TRUE)
Projects are the core building blocks of the platform. Each project corresponds to a distinct scientific investigation, serving as a container for its data, analysis tools, results, and collaborators.
Create a new project called “api testing” with the billing group id
obtained above.
# get billing group id bid <- a$billing()$id # create new project (p <- a$project_new(name = "api testing", bid, description = "Just a test"))
== Project ==
id : RFranklin/api-testing
name : api testing
description : Just a test
billing_group_id : <your_bid>
type : v2
-- Permission --
# list first 100 a$project() # list all a$project(complete = TRUE) # return all named match "demo" a$project(name = "demo", complete = TRUE) # get the project you want by id p <- a$project(id = "RFranklin/api-tutorial")
Seven Bridges maintains workflows and tools available to all of its users in the Public Apps repository.
To find out more about public apps, you can do the following:
sevenbridges
package to find it, as shown below.# search by name matching, complete = TRUE search all apps, # not limited by offset or limit. a$public_app(name = "STAR", complete = TRUE) # search by id is accurate a$public_app(id = "admin/sbg-public-data/rna-seq-alignment-star/5") # you can also get everything a$public_app(complete = TRUE) # default limit = 100, offset = 0 which means the first 100 a$public_app()
Now, from your Auth
object, you copy an App id
into your project
id with a new name
, following this logic.
# copy a$copy_app( id = "admin/sbg-public-data/rna-seq-alignment-star/5", project = "RFranklin/api-testing", name = "new copy of star" ) # check if it is copied p <- a$project(id = "RFranklin/api-testing") # list apps your got in your project p$app()
The short name is changed to newcopyofstar
.
== App ==
id : RFranklin/api-testing/newcopyofstar/0
name : RNA-seq Alignment - STAR
project : RFranklin/api-testing-2
revision : 0
Alternatively, you can copy it from the app
object.
app <- a$public_app(id = "admin/sbg-public-data/rna-seq-alignment-star") app$copy_to( project = "RFranklin/api-testing", name = "copy of star" )
You can also upload your own Common Workflow Language JSON file which describes your app to your project.
Note: Alternatively, you can directly describe your CWL tool in R with this package. Please read the vignette on “Describe CWL Tools/Workflows in R and Execution”.
# add an CWL file to your project f.star <- system.file("extdata/app", "flow_star.json", package = "sevenbridges") app <- p$app_add("starlocal", fl.runif) (aid <- app$id)
You will get an app id
, like the one below:
"RFranklin/api-testing/starlocal/0"
It’s composed of the following elements:
RFranklin/api
runif
0
Alternatively, you can describe tools in R directly, as shown below:
fl <- system.file("docker", "sevenbridges/rabix/generator.R", package = "sevenbridges" ) cat(readLines(fl), sep = "\n")
library("sevenbridges")
in.lst <- list(
input(
id = "number",
description = "number of observations",
type = "integer",
label = "number",
prefix = "--n",
default = 1,
required = TRUE,
cmdInclude = TRUE
),
input(
id = "min",
description = "lower limits of the distribution",
type = "float",
label = "min",
prefix = "--min",
default = 0
),
input(
id = "max",
description = "upper limits of the distribution",
type = "float",
label = "max",
prefix = "--max",
default = 1
),
input(
id = "seed",
description = "seed with set.seed",
type = "float",
label = "seed",
prefix = "--seed",
default = 1
)
)
# the same method for outputs
out.lst <- list(
output(
id = "random",
type = "file",
label = "output",
description = "random number file",
glob = "*.txt"
),
output(
id = "report",
type = "file",
label = "report",
glob = "*.html"
)
)
rbx <- Tool(
id = "runif",
label = "Random number generator",
hints = requirements(docker(pull = "tengfei/runif"), cpu(1), mem(2000)),
baseCommand = "runif.R",
inputs = in.lst, # or ins.df
outputs = out.lst
)
fl <- "inst/docker/sevenbridges/rabix/runif.json"
write(rbx$toJSON(pretty = TRUE), fl)
Then, you can add it like this:
# rbx is the object returned by `Tool` function app <- p$app_add("runif", rbx) (aid <- app$id)
Please consult another tutorial vignette("apps", "sevenbridges")
about how to describe tools and flows in R.
Once you have copied the public app admin/sbg-public-data/rna-seq-alignment-star/5
into your project, username/api-testing
, the app id
in your current project is username/api-testing/newcopyofstar
. Conversely, you can use another app you already have in your project for this Quickstart.
To draft a new task, you need to specify the following:
id
of the workflow you are executingYou can always check the App details on the visual interface for task input requirements. To find the required inputs with R, you need to get an App
object first.
app <- a$app(id = "RFranklin/api-testing-2/newcopyofstar") # get input matrix app$input_matrix() app$input_matrix(c("id", "label", "type")) # get required node only app$input_matrix(c("id", "label", "type"), required = TRUE)
Conversely, you can load the app from a CWL JSON and convert it into an R object first, as shown below.
f1 <- system.file("extdata/app", "flow_star.json", package = "sevenbridges") app <- convert_app(f1) # get input matrix app$input_matrix()
## id label type required
## 1 #sjdbGTFfile sjdbGTFfile File... FALSE
## 2 #fastq fastq File... TRUE
## 3 #genomeFastaFiles genomeFastaFiles File TRUE
## 4 #sjdbGTFtagExonParentTranscript Exons' parents name string FALSE
## 5 #sjdbGTFtagExonParentGene Gene name string FALSE
## 6 #winAnchorMultimapNmax Max loci anchors int FALSE
## 7 #winAnchorDistNbins Max bins between anchors int FALSE
## fileTypes
## 1 null
## 2 null
## 3 null
## 4 null
## 5 null
## 6 null
## 7 null
app$input_matrix(c("id", "label", "type"))
## id label type
## 1 #sjdbGTFfile sjdbGTFfile File...
## 2 #fastq fastq File...
## 3 #genomeFastaFiles genomeFastaFiles File
## 4 #sjdbGTFtagExonParentTranscript Exons' parents name string
## 5 #sjdbGTFtagExonParentGene Gene name string
## 6 #winAnchorMultimapNmax Max loci anchors int
## 7 #winAnchorDistNbins Max bins between anchors int
app$input_matrix(c("id", "label", "type"), required = TRUE)
## id label type
## 2 #fastq fastq File...
## 3 #genomeFastaFiles genomeFastaFiles File
Note that input_matrix
and output_matrix
are useful accessors for Tool
, Flow
, you can call these two function on a JSON file directly as well.
tool.in <- system.file("extdata/app", "tool_unpack_fastq.json", package = "sevenbridges") flow.in <- system.file("extdata/app", "flow_star.json", package = "sevenbridges") input_matrix(tool.in) input_matrix(tool.in, required = TRUE) input_matrix(flow.in) input_matrix(flow.in, c("id", "type")) input_matrix(flow.in, required = TRUE) output_matrix(tool.in) output_matrix(flow.in)
In the response body, locate the names of the required inputs. Note that task inputs need to match the expected data type and name. In the above example, we see two required fields:
We also want to provide a gene feature file:
You can find a list of possible input types below:
File
), other inputs take more than one file (File
arrays, FilesList
, or ‘File...
’ ). This input requires you to pass a Files
object (for a single file input) or FilesList
object (for inputs which accept more than one file) or simply a list in a “Files” object. You can search for your file by id
or by name
with an exact match (exact = TRUE
), as shown in the example below.fastqs <- c("SRR1039508_1.fastq", "SRR1039508_2.fastq") # get all 2 exact files fastq_in <- p$file(name = fastqs, exact = TRUE) # get a single file fasta_in <- p$file( name = "Homo_sapiens.GRCh38.dna.primary_assembly.fa", exact = TRUE ) # get all single file gtf_in <- p$file( name = "Homo_sapiens.GRCh38.84.gtf", exact = TRUE )
# add new tasks taskName <- paste0("RFranklin-star-alignment ", date()) tsk <- p$task_add( name = taskName, description = "star test", app = "RFranklin/api-testing-2/newcopyofstar/0", inputs = list( sjdbGTFfile = gtf_in, fastq = fastq_in, genomeFastaFiles = fasta_in ) )
Remember the fastq
input expects a list of files. You can also do something as follows:
f1 <- p$file(name = "SRR1039508_1.fastq", exact = TRUE) f2 <- p$file(name = "SRR1039508_2.fastq", exact = TRUE) # get all 2 exact files fastq_in <- list(f1, f2) # or if you know you only have 2 files whose names match SRR924146*.fastq fastq_in <- p$file(name = "SRR1039508*.fastq", complete = TRUE)
Use complete = TRUE
when the number of items is over 100.
Now let’s do a batch with 8 files in 4 groups, which is batched by metadata sample_id
and library_id
. We will assume each file has these two metadata fields entered. Since these files can be evenly grouped into 4, we will have a single parent batch task with 4 child tasks.
fastqs <- c( "SRR1039508_1.fastq", "SRR1039508_2.fastq", "SRR1039509_1.fastq", "SRR1039509_2.fastq", "SRR1039512_1.fastq", "SRR1039512_2.fastq", "SRR1039513_1.fastq", "SRR1039513_2.fastq" ) # get all 8 files fastq_in <- p$file(name = fastqs, exact = TRUE) # can also try to returned all SRR*.fastq files # fastq_in <- p$file(name= "SRR*.fastq", complete = TRUE) tsk <- p$task_add( name = taskName, description = "Batch Star Test", app = "RFranklin/api-testing-2/newcopyofstar/0", batch = batch( input = "fastq", criteria = c("metadata.sample_id", "metadata.noexist_id") ), inputs = list( sjdbGTFfile = gtf_in, fastq = fastqs_in, genomeFastaFiles = fasta_in ) )
Now you have a draft batch task. Please check it out in the visual interface. Your response body should inform you of any errors or warnings.
Note: you can also directly pass file id or file names as characters to inputs list, the package will first guess if the passed strings are file id (24-bit hexadecimal) or names, then convert it to Files or FilesList object. However, as a good practice, we recommend you construct your files object(e.g. p$file(id = ..., name = ...)
) first, check the value, then pass it to task_add
inputs. This is a safer approach.
 Alternatively, you can delete the draft task if you no longer wish to run it.
# # not run # tsk$delete()
After you run a task, you can abort it.
# abort your task tsk$abort()
If you want to update your task and then re-run it, follow the example below.
tsk$getInputs() # missing number input, only update number tsk$update(inputs = list(sjdbGTFfile = "some new file")) # double check tsk$getInputs()
Running tasks with spot instances could potentially reduce a considerable amount of computational cost. This option can be controlled on the project level or the task level on Seven Bridges platforms. Our package follows the same logic as our platform’s web interface (the current default setting for spot instances is on).
For example, when we create a project using project_new()
, we can set use_interruptible = FALSE
to use on-demand instances (non-interruptible but more expensive) instead of the spot instances (interruptible but cheaper):
p <- a$project_new( name = "spot-disabled-project", bid, description = "spot disabled project", use_interruptible = FALSE )
Then all the new tasks created under this project will use on-demand instances to run by default, unless an argument use_interruptible_instances
is specifically set to TRUE
when drafting the new task using task_add()
.
For example, if p
is the above spot disabled project, to draft a task that will use spot instances to run:
tsk <- p$task_add( name = paste0("spot enabled task in a spot disabled project"), description = "spot enabled task", app = ..., inputs = list(...), use_interruptible_instances = TRUE )
Conversely, you can have a spot instance enabled project, but draft and run specific tasks using on-demand instances, by setting use_interruptible_instances = FALSE
in task_add()
explicitly.
During workflow development and benchmarking, sometimes we need to view and make adjustments to the computational resources needed for a task to run more efficiently. Also, if a task fails due to resource deficiency, we often want to define a larger instance for the task re-run without editing the app itself. This is particularly important in cases where there is not enough disk space.
The Seven Bridges API allows setting specific task execution parameters by using execution_settings
. It includes the instance type (instance_type
) and the maximum number of parallel instances (max_parallel_instances
):
tsk <- p$task_add( ..., execution_settings = list( instance_type = "c4.2xlarge;ebs-gp2;2000", max_parallel_instances = 2 ) )
For details about execution_settings
, please check create a new draft task.
To monitor your task as it runs, you can always request a task update
to ask your task to report its status. Or, you can monitor a running task with a hook function, which triggers the function when that status is “completed”. Please check the details in section below.
tsk$update()
By default, your task alerts you by email when it has been completed.
# Monitor your task (skip this part) # tsk$monitor()
Use the following to download all files from a completed task.
tsk$download("~/Downloads")
Instead of the default task monitor action, you can use setTaskHook
to connect a function call to the status of a task. When you run tsk$monitor(time = 30)
it will check your task every 30 seconds to see if the current task status matches one of the following statuses: “queued”, “draft”, “running”, “completed”, “aborted”, and “failed”. When it finds a match for the task status, getTaskHook
returns the function call for the specific status.
getTaskHook("completed")
## function(...) {
## cat("\r", "completed")
## return(TRUE)
## }
## <environment: 0x7f9820493fd0>
If you want to customize the monitor function, you can adjust the following requirement. Your function must return TRUE
or FALSE
in the end. When it is TRUE
(or non-logical value) it means the monitoring will be terminated after it finds a status matched and the function executes, such as when the task is completed. When it is FALSE
, it means the monitoring will continue for the next iteration of checking, e.g., when it is “running”, you want to keep tracking.
Follow the example below to set a new function to monitor the status “completed”. Then, when the task is completed, it will download all task output files to the local folder.
setTaskHook("completed", function() { tsk$download("~/Downloads") return(TRUE) }) tsk$monitor()
The sevenbridges
package provides a user-friendly interface, so you do not have to combine several api()
calls and constantly reference the API documentation to issue API requests.
Before you can interact with the API, you need to construct an Auth
object which stores the following information:
The general authentication logic for Auth()
is as follows:
from
is not specified explicitly or specified as from = "direct"
.from = "env"
, or user configuration file when from = "file"
.To use direct authentication, users need to specify one of platform
or url
, with the corresponding token
. Examples of direct authentication:
a <- Auth( token = "your_token", platform = "aws-us" )
The above will use the Seven Bridges Platform on AWS (US).
a <- Auth( token = "your_token", url = "https://gcp-api.sbgenomics.com/v2" )
The above will use the specified url
as the base URL for the API calls. In this example, the url
points to the Seven Bridges Platform on Google Cloud Platform (GCP).
a <- Auth(token = "your_token")
The above will use the Cancer Genomics Cloud environment since no platform
nor url
were explicitly specified (not recommended).
Note: platform
and url
should not be specified at the same time.
The R API client supports reading authentication information stored in system environment variables.
To set the two environment variables in your system, you could use the function sbg_set_env()
. For example:
sbg_set_env( url = "https://cgc-api.sbgenomics.com/v2", token = "your_token" )
See if the environment variables are correctly set:
sbg_get_env("SB_API_ENDPOINT") ## "https://cgc-api.sbgenomics.com/v2" sbg_get_env("SB_AUTH_TOKEN") ## "your_token"
To create an Auth
object:
a <- Auth(from = "env")
To unset the two environment variables:
Sys.unsetenv("SB_API_ENDPOINT") Sys.unsetenv("SB_AUTH_TOKEN")
You can create an ini-like file named credentials
under the folder $HOME/.sevenbridges/
and maintain your credentials for multiple accounts across various Seven Bridges environments. An example:
[aws-us-rfranklin]
api_endpoint = https://api.sbgenomics.com/v2
auth_token = token_for_this_user
# This is a comment:
# another user on the same platform
[aws-us-rosalind-franklin]
api_endpoint = https://api.sbgenomics.com/v2
auth_token = token_for_this_user
[default]
api_endpoint = https://cgc-api.sbgenomics.com/v2
auth_token = token_for_this_user
[gcp]
api_endpoint = https://gcp-api.sbgenomics.com/v2
auth_token = token_for_this_user
Please make sure to have two fields exactly named as api_endpoint
and auth_token
under each profile.
To load the default profile (named [default]
) from the default user configuration file ($HOME/.sevenbridges/credentials
), please use:
a <- Auth(from = "file")
To load the user profile aws-us-rfranklin
from this configuration file, change the profile_name
:
a <- Auth(from = "file", profile_name = "aws-us-rfranklin")
To use a user configuration file from other locations (not recommended), please specify the file path using the argument config_file
. For example:
a <- Auth( from = "file", config_file = "~/sevenbridges.cfg", profile_name = "aws-us-rfranklin" )
Note: If you edited the credentials
file, please use Auth()
to re-authenticate.
If you did not pass any parameters to api()
from Auth
, it would list all API calls. Any parameters you provide will be passed to the api()
function, but you do not have to pass your input token and path once more since the Auth
object already has this information. The following call from the Auth
object will check the response as well.
a$api()
offset
and limit
Every API call accepts two arguments named offset
and limit
.
offset
defines where the retrieved items started.limit
defines the number of items you want to get.By default, offset
is set to 0
and limit
is set to 50
. As such, your API request returns the first 100 items when you list items or search for items by name. To search and list all items, use complete = TRUE
in your API request.
When searching by ID, your request will return your exact resource as it is unique. As such, you do not have to set offset
and limit
manually. It is a good practice to find your resources by their ID and pass this ID as an input to your task. You can find a resource’s ID in the final part of the URL on the visual interface or via the API requests to list resources or get a resource’s details.
Search by name returns all partial matches unless you specify exact = TRUE
. This type of search will only search across current pulled content, so use complete = TRUE
if you want to search across everything.
For example, to list all public apps, use visibility
argument, but make sure you pass complete = TRUE
to it, to show everything. This arguments generally works for items like “App”, “Project”, “Task”, “File”, etc.
# first, search by id is fast x <- a$app(visibility = "public", id = "admin/sbg-public-data/sbg-ucsc-b37-bed-converter/1") # show 100 items from public x <- a$app(visibility = "public") length(x) # 100 x <- a$app(visibility = "public", complete = TRUE) length(x) # 272 by Nov 2016 # this return nothing, because it is not in the first 100 returned names a$app(visibility = "public", name = "bed converter") # this return an app, because it pulls *all* app names and did search a$app(visibility = "public", name = "bed converter", complete = TRUE)
Similar to offset
and limit
, every API call accepts an argument named advance_access
. This argument was first introduced in August 2017 and controls if a special field in the HTTP request header will be sent, which can enable the access to the “Advance Access” features in the Seven Bridges API. Note that the Advance Access features in the API are not officially released yet, therefore the API usages are subject to change, so please use with discretion.
In addition to modifying each API call that uses Advance Access features, the option can also be set globally at the beginning of your API script. This offers a one-button switch for users who want to experiment with the Advance Access features. The option is disabled by default:
library("sevenbridges") getOption("sevenbridges")$advance_access
## [1] FALSE
For example, the Markers API is in Advance Access as of November 2018. If we try to use the Markers API to list markers available on a BAM file with the advance_access
option disabled, it will return an error message:
req <- api( token = "your_token", path = "genome/markers?file={bam_file_id}", method = "GET" ) httr::content(req)$"message"
## [1] "Advance access feature needs X-SBG-Advance-Access: advance header."
To enable the Advance Access features, one can use
at the beginning of their scripts. Let’s check if the option has been enabled:
getOption("sevenbridges")$advance_access
## [1] TRUE
Send the API call again:
req <- api( token = "your_token", path = "genome/markers?file={bam_file_id}", method = "GET" )
The information of the markers available on that BAM file will be returned:
httr::content(req)
$href
[1] "https://api.sbgenomics.com/v2/genome/markers?file={bam_file_id}&offset=0&limit=2"
$items
$items[[1]]
$items[[1]]$href
[1] "https://api.sbgenomics.com/v2/genome/markers/{bam_file_id}"
$items[[1]]$id
[1] "{bam_file_id}"
$items[[1]]$name
[1] "Marker Title"
$items[[2]]
$items[[2]]$href
[1] "https://api.sbgenomics.com/v2/genome/markers/{bam_file_id}"
$items[[2]]$id
[1] "{bam_file_id}"
$items[[2]]$name
[1] "Marker Title"
$links
list()
'fields'
Please read the documentation detail.
All API calls take the optional query parameter fields. This parameter enables you to specify the fields you want to be returned when listing resources (e.g., all your projects) or getting details of a specific resource (e.g., a given project).
The fields parameter can be used in the following ways:
No fields
parameter specified: calls return default fields. For calls that return complete details of a single resource, this is all their properties; for calls that list resources of a certain type, this is some default properties.
The fields
parameter can be set to a list of fields: for example, to return the fields id, name and size for files in a project, you may issue the call p$file(fields = "id,name,size")
The fields parameter can be used to exclude a specific file: if you wish to omit certain field from the response, do so using the fields parameter with the prefix !
. For example, to get the details of a file without listing its metadata, issue a call p$file(fields = '!metadata')
The entire metadata field will be removed from the response.
The fields parameter can be used to include or omit certain nested fields, in the same way as listed in 2 and 3 above: for example, you can use metadata.sample_id
or origin.task
for files.
To see all fields for a resource, specify fields="_all"
. This returns all fields for each resource returned. Note that if you are getting the details of a specific resource, the use of fields="_all"
won’t return any more properties than would have been shown without this parameter — the use case is instead for when you are listing details of many resources. Please use with care if your resource has particularly large fields; for example, the raw field for an app resource contains the complete CWL specification of the app which can result in bulky response if listing many apps.
Negations and nesting can be combined freely, so, for example, you can issue p$file(fields="id,name,status,!metadata.library,!origin")
or p$task(fields="!inputs,!outputs")
.
Please try the following examples.
## default fields id, name, project p$file() ## return file(s) with id, name, siae information p$file(fields = "id,name,size") ## return file(s) with all available info p$file(detail = TRUE) ## same as above p$file(fields = "_all")
This call returns information about your current rate limit. This is the number of API calls you can make in one hour.
a$rate_limit()
This call returns a list of the resources, such as projects, billing groups, and organizations, that are accessible to you. Currently, this call will only return a successful response if {username} is replaced with your own username. Be sure to capitalize your username in the same way as when you registered for an account.
If you did not provide a username, it will show your user information.
# return your information a$user() # return user RFranklin's information a$user("RFranklin")
If no billing group id
is provided, this call returns a list of paths used to access billing information via the API. If a username is provided, this call lists all your billing groups, including groups that are pending or which have been disabled. If you specify breakdown = TRUE
, the call below returns a breakdown of the spending per-project for the billing group specified by billing_group. Information is also displayed for each of the projects a particular billing group is associated with, including task executions, their initiating user, start and end time, and their cost.
# return a BillingList object (b <- a$billing()) a$billing(id = b$id, breakdown = TRUE)
If no Billing Group id
provided, This call returns a list of invoices, with information about each, including whether or not the invoice is pending and the billing period it covers. This call returns information about all your available invoices unless you use the query parameter bg_id to specify the ID of a particular Billing Group, in which case it will return the invoice incurred by that Billing Group only. If an invoice id
is provided, this call retrieves information about the specified invoice, including the costs for analysis and storage and the invoice period.
a$invoice() a$invoice(id = "your_id")
Note: Currently, invoice is not an object. Instead, it just returns a regular R list.
Projects are the basic units to organize different entities: files, tasks, apps, etc. As such, many actions stem from the Project
object.
The following call returns a list of all projects of which you are a member. Each project’s project_id
and the path will be returned.
a$project()
If you want to list the projects owned by and accessible to a particular user, specify the owner
argument as follows. Each project’s ID and path will be returned.
a$project(owner = "RFranklin") a$project(owner = "Rosalind.Franklin")
To get details about the project(s), use detail = TRUE
, as shown below.
a$project(detail = TRUE)
For a more friendly interface and convenient search, the sevenbridges
package supports partial name matching. The first argument for the following request is name
.
# want to return a project called a$project("hello")
Project creation date, modification date, and creator information is useful for quickly locating the project you need, especially when you want to follow the life cycle of a large number of projects and distinguish recent projects from old ones. To facilitate such needs, the fields created_by
, created_on
, and modified_on
are returned in the project query calls:
# return all projects matching the name "wgs" p <- a$project("wgs", complete = TRUE) # filter by project creators creators <- sapply(p, "[[", "created_by") which(creator == "RFranklin") # filter by project creation date create_date <- as.Date(sapply(p, "[[", "created_on")) which(as.Date(create_date) < as.Date("2019-01-01")) # filter by project modification date modify_date <- as.Date(sapply(p, "[[", "modified_on")) which(as.Date(modify_date) < as.Date("2019-01-01"))
To create a new project, users need to specify the following:
name
(required)billing_group_id
(required)description
(optional)tags
(optional): This has to be a list(). If you are using the API on the CGC environment, you can create a TCGA Controlled Data project by specifying TCGA
in tags
.type
(optional): By default, we create a V2, CWL compatible project.a$project_new("api_testing_tcga", b$id, description = "Test for API" )
Follow the directions above, but pass tcga
as a value for tags
.
a$project_new("controlled_project", b$id, description = "Test for API", tags = list("tcga") )
You can delete a single project by making the request to $delete()
. Note that the returned object from a$project()
sometimes returns list if you use partial matching by name. The $delete()
request cannot operate on a list. If you want to operate on a list of object, read more about batch functions in the relevant section below.
# remove it, not run a$project("api_testing")$delete() # check # will delete all projects match the name delete(a$project("api_testing_donnot_delete_me"))
You can update the following information about an existing project:
name
description
billing_group_id
a$project(id = "RFranklin/helloworld") a$project(id = "RFranklin/helloworld")$update( name = "Hello World Update", description = "Update description" )
This call returns a list of the members of the specified project. For each member, the response lists:
a$project(id = "RFranklin/demo-project")$member()
This call adds a new user to the specified project. It can only be made by a user who has admin permissions in the project.
Requests to add a project member must include the key permissions
. However, if you do not include a value, the member’s permissions will be set to false
by default.
Set permission by passing the following values: copy
, write
, execute
, admin
, or read
.
Note: read
is implicit and set by default. You can not be a project member without having read
permissions.
m <- a$project(id = "RFranklin/demo-project")$ member_add(username = "Rosalind.Franklin")
This call edits a user’s permissions in the specified project. It can only be made by a user who has admin permissions in the project.
m <- a$project(id = "RFranklin/demo-project")$ member(username = "Rosalind.Franklin") m$update(copy = TRUE)
== Member ==
username : Rosalind.Franklin
-- Permission --
read : TRUE
write : FALSE
copy_permission : TRUE
execute : FALSE
admin : FALSE
The Folders API allows you to organize your files using folders in projects. From a filesystem perspective, folders are simply a special type of file. Therefore, within this package, the folder objects are still represented using the same class as the file objects above, only with specific methods for operating on them.
To create new folders and manage files using folders in a project, we first need to get the information about the root folder of the project. For example
p <- a$project(id = "RFranklin/folders-api-testing")
We can get the project root folder ID:
p$get_root_folder_id()
[1] "5bd7c53ee4b04b8fb1a9f63c"
Or, return the project root folder as an object:
(root_folder <- p$get_root_folder())
== Files ==
id : 5bd7c53ee4b04b8fb1a9f63c
name : 8c02ceda-e46b-49c0-362f-4318fa8f3e4k
project : RFranklin/folders-api-testing
created_on : 2018-10-29T18:45:50Z
modified_on : 2018-10-29T18:45:50Z
type : folder
To create a new_folder under the project root folder, use
folder1 <- root_folder$create_folder("folder1")
To create a new folder under folder1
, use
folder2 <- folder1$create_folder("folder2")
Let’s copy a public reference file into the project root folder:
file_public <- a$public_file(name = "Homo_sapiens_assembly38.fasta", exact = TRUE) file1 <- file_public$copy_to_folder(root_folder$id)
We can also rename the file when copying it into a folder:
file2 <- file_public$copy_to_folder(root_folder$id, "ref_grch38.fasta")
To move files to another folder (for example, folder1
we just created), use
file1 <- file1$move_to_folder(folder1$id)
To list all files and folders in a folder (for example, the root folder), use
(contents_root <- root_folder$list_folder_contents(complete = TRUE))
[[1]]
== Files ==
id : 5bede476e4b03b8ff38c28ba
name : folder1
project : RFranklin/folders-api-testing
parent : 5bd7c53ee4b04b8fb1a9f63c
type : folder
[[2]]
== Files ==
id : 5bedc431e4b09b8ff3966e9c
name : ref_grch38.fasta
project : RFranklin/folders-api-testing
parent : 5bd7c53ee4b04b8fb1a9f63c
type : file
If complete = FALSE
(default), it will only return the (first) page from the API based on offset
and limit
.
To list only files or only folders, set type = "file"
or type = "folder"
:
root_folder$list_folder_contents(type = "file", complete = TRUE) root_folder$list_folder_contents(type = "folder", complete = TRUE)
Get the type of a single file/folder object:
contents_root[[1]]$typeof()
[1] "folder"
Or, get the type(s) for all objects in the folder:
sapply(contents_root, function(x) x$typeof())
[1] "folder" "file"
Sometimes, it’s convenient to get the parent folder ID for a file or folder:
contents_root[[1]]$get_parent_folder_id()
[1] "5bd7c53ee4b04b8fb1a9f63c"
This is essentially the root folder ID. Alternatively, to get the parent folder as an object, use
contents_root[[1]]$get_parent_folder()
If a folder still has files in it, it cannot be deleted:
file2 <- file2$move_to_folder(folder2$id) folder2$delete()
Error: HTTP Status 400: Deleting non-empty folders is not supported.
We need to delete all files (and subfolders) within a folder before deleting the folder itself:
file2$delete() folder2$delete() folder1$delete()
All Seven Bridges environments support the Common Workflow Language (CWL) natively to allow for reproducible and portable workflows and tools. In this section, we will work with apps, including tools and workflows, via the API using the sevenbridges
R package.
This call lists all the apps available to you.
a$app() # or show details a$app(detail = TRUE)
Search for an app by name
To search for a specific app by its name, pass a pattern for the name
argument or provide a unique id
as shown below.
# pattern match a$app(name = "STAR") # unique id aid <- a$app()[[1]]$id aid a$app(id = aid) # get a specific revision from an app a$app(id = aid, revision = 0)
List all apps in a project
To list all apps belonging to a specific project, use the project
argument.
# my favorite, always a$project("demo")$app() # or alternatviely pid <- a$project("demo")$id a$app(project = pid)
List all public apps
To list all public apps, use the visibility
argument.
# show 100 items from public x <- a$app(visibility = "public") length(x) x <- a$app(visibility = "public", complete = TRUE) length(x) x <- a$app(project = "RFranklin/helloworld", complete = TRUE) length(x) a$app(visibility = "public", limit = 5, offset = 150)
Search through all public apps in all locations
To search for a public app cross all locations, make the following call. Note that this may take a bit of time.
a$app("STAR", visibility = "public", complete = TRUE)
This call copies the specified app to a specified project. The app should be one in a project that you can access, either an app which has been uploaded by a project member or a publicly available app which has been copied to the project.
This call requires the following two arguments:
project
: include the project idname
(optional): use this field to optionally re-name your appaid <- a$public_app()[[1]]$id a$copy_app(aid, project = pid, name = "copy-rename-test") # check if it is copied a$app(project = pid)
You can also copy directly from the app object, as shown below.
app <- a$public_app(id = "admin/sbg-public-data/rna-seq-alignment-star") app$copy_to( project = "RFranklin/api-testing", name = "copy of star" )
This call returns the raw CWL of a specific app, including its raw CWL. The call differs from the call to GET details of an app in that it returns a JSON object that is the CWL.
The app should be one in a project that you can access, either an app which has been uploaded by a project member or a publicly available app which has been copied to the project.
To get a specific revision, pass the revision
argument.
ap <- a$app(visibility = "public")[[1]] a$project("demo")$app("index") # get a specific revision a$project("demo")$app("index", revision = 0)
Coming soon: converting apps to CWL objects
To add a CWL object as an app, use the app_add
function call for a Project
object. The following two parameters are required:
short_name
: Project short names are based on the name you give to a project when you create it. Learn more about project short names on the Seven Bridges Platform or the CGC.filename
: The name of the JSON file containing the CWL.cwl.fl <- system.file("extdata", "bam_index.json", package = "sevenbridges") a$project("demo")$app_add(short_name = "new_bam_index_app", filename = cwl.fl) a$project("demo")$app_add(short_name = "new_bam_index_app", revision = 2, filename = cwl.fl)
Note: If you provide the same short_name
, this will add a new revision.
This is introduced in another vignette (vignette("apps", "sevenbridges")
).
On Seven Bridges platforms, when you create or update your tools in the GUI, there is a test tab allow users to tweak the parameters and see what it looks like in your command line simulated terminal. To do this via R when you push your Tool object to your project, you need to provide "sbg:job"
information like the example shown below
rbx <- Tool( id = "runif", label = "Random number generator", hints = requirements( docker(pull = "RFranklin/runif"), cpu(1), mem(2000) ), baseCommand = "runif.R", inputs = in.lst, outputs = out.lst, "sbg:job" = list( allocatedResources = list(mem = 9000, cpu = 1), inputs = list(min = 1, max = 150) ) ) p$app_add("random", rbx)
Or if you have created test info on the platform or previously pushed one, and you want to keep the previous test setup. We provide an argument named keep_test
to allow you to keep the previous revision’s test information.
rbx <- Tool( id = "runif", label = "Random number generator", hints = requirements( docker(pull = "RFranklin/runif"), cpu(1), mem(2000) ), baseCommand = "runif.R", inputs = in.lst, outputs = out.lst ) p$app_add("random", rbx, keep_test = TRUE)
This call returns a list of tasks that you can access. You can filter tasks by status.
# all tasks a$task() # filter a$task(status = "completed") a$task(status = "running")
To list all the tasks in a project, use the following.
# a better way a$project("demo")$task() # alternatively pid <- a$project("demo")$id a$task(project = pid)
To list all tasks with details, pass detail = TRUE
.
p$task(id = "your task id here", detail = TRUE) p$task(detail = TRUE)
To list a batch task using the parent
parameter, pass the batch parent task id.
p <- a$project(id = "RFranklin/demo") p$task(id = "2e1ebed1-c53e-4373-870d-4732acacbbbb") p$task(parent = "2e1ebed1-c53e-4373-870d-4732acacbbbb") p$task(parent = "2e1ebed1-c53e-4373-870d-4732acacbbbb", status = "completed") p$task(parent = "2e1ebed1-c53e-4373-870d-4732acacbbbb", status = "draft")
To create a draft task, you need to call the task_add
function from the Project object. And you need to pass the following arguments:
name
: The name for this taskdescription
(optional): A description for this taskapp
: The app idinputs
: A list of inputs for this task# push an app first fl.runif <- system.file("extdata", "runif.json", package = "sevenbridges") a$project("demo")$app_add("runif_draft", fl.runif) runif_id <- "RFranklin/demo-project/runif_draft" # create a draft task a$project("demo")$task_add( name = "Draft runif 3", description = "Description for runif 3", app = runif_id, inputs = list(min = 1, max = 10) ) # confirm a$project("demo")$task(status = "draft")
Call the update
function from a Task object to update the following:
name
description
inputs
list. Note that you can only update the items you provided.# get the single task you want to update tsk <- a$project("demo")$task("Draft runif 3") tsk tsk$update( name = "Draft runif update", description = "draft 2", inputs = list(max = 100) ) # alternative way to check all inputs tsk$getInputs()
This call runs (“executes”) the specified task. Only tasks with a “DRAFT” status may be run.
tsk$run() # run update without information just return latest information tsk$update()
### Monitor a running task and set the hook function
To monitor a running task, call monitor
from a Task object.
tsk$monitor()
Get and set default a hook function for task status. Currently, failed tasks will break the monitoring.
Note: Hook function has to return TRUE
(break monitoring) or FALSE
(continuing) in the end.
getTaskHook("completed") getTaskHook("draft") setTaskHook("draft", function() { message("never happens") return(TRUE) }) getTaskHook("draft")
This call aborts the specified task. Only tasks whose status is “RUNNING” may be aborted.
# abort tsk$abort() # check tsk$update()
Note that you can only delete DRAFT tasks, not running tasks.
tsklst <- a$task(status = "draft") # delete a single task tsklst[[1]]$delete() # confirm a$task(status = "draft") # delete a list of tasks delete(tsklst)
To run tasks in batch mode, (check ?batch
) for more details. The code below is a mockup example.
# batch by items (tsk <- p$task_add( name = "RNA DE report new batch 2", description = "RNA DE analysis report", app = rna.app$id, batch = batch(input = "bamfiles"), inputs = list( bamfiles = bamfiles.in, design = design.in, gtffile = gtf.in ) )) # batch by metadata, input files has to have metadata fields specified (tsk <- p$task_add( name = "RNA DE report new batch 3", description = "RNA DE analysis report", app = rna.app$id, batch = batch( input = "fastq", c("metadata.sample_id", "metadata.library_id") ), inputs = list( bamfiles = bamfiles.in, design = design.in, gtffile = gtf.in ) ))
Cloud storage providers come with their own interfaces, features, and terminology. At a certain level, though, they all view resources as data objects organized in repositories. Authentication and operations are commonly defined on those objects and repositories, and while each cloud provider might call these things different names and apply different parameters to them, their basic behavior is the same.
Seven Bridges environments mediate access to these repositories using volumes. A volume is associated with a particular cloud storage repository that you have enabled Seven Bridges to read from (and, optionally, to write to). Currently, volumes may be created using two types of cloud storage repository: Amazon Web Services’ (AWS) S3 buckets and Google Cloud Storage (GCS) buckets.
A volume enables you to treat the cloud repository associated with it as external storage. You can ‘import’ files from the volume to your Seven Bridges environment to use them as inputs for computation. Similarly, you can write files from the Seven Bridges environment to your cloud storage by ‘exporting’ them to your volume.
Learn more about volumes on the Seven Bridges Platform and the CGC.
a <- Auth(user = "RFranklin", platform = "aws-us") a$add_volume( name = "tutorial_volume", type = "s3", bucket = "RFranklin-demo", prefix = "", access_key_id = "your_access_key_id", secret_access_key = "your_secret_access_key", sse_algorithm = "AES256", access_mode = "RW" )
# list all volume a$volume() # get unique volume by id a$volume(id = "RFranklin/RFranklin_demo") # partial search by name a$volume(name = "demo")
This call deletes a volume you’ve created to refer to storage on Amazon Web Services or Google Cloud Storage.
Note that any files you’ve imported from your volume onto a Seven Bridges environment, known as an alias, will no longer be usable. If a new volume is created with the same volume_id as the deleted volume, aliases will point to files on the newly created volume instead (if those exist).
a$volume(id = "RFranklin/RFranklin_demo")$delete()
This call imports a file from volume to your project.
v <- a$volume(id = "RFranklin/tutorial_volume") res <- v$import( location = "A-RNA-File.bam.bai", project = "RFranklin/s3tutorial", name = "new.bam.bai", overwrite = TRUE ) # get job status update # state will be "COMPLETED" when it's finished, otherwise "PENDING" v$get_import_job(res$id) v
Important :
When testing, please update your file in a project.
res <- v$export( file = "579fb1c9e4b08370afe7903a", volume = "RFranklin/tutorial_volume", location = "", # when "" use old name sse_algorithm = "AES256" ) # get job status update # state will be "COMPLETED" when it's finished other wise "PENDING" v$get_export_job(res$id) v
Seven Bridges hosts publicly accessible files and apps on its environments. The sevenbridges
package provides two easy function calls from the Authentication object to search for and copy files and apps to a project.
# list the first 100 files a$public_file() # list by offset and limit a$public_file(offset = 100, limit = 100) # simply list everything! a$public_file(complete = TRUE) # get exact file by id a$public_file(id = "5772b6f0507c175267448700") # get exact file by name with exact = TRUE a$public_file(name = "G20479.HCC1143.2.converted.pe_1_1Mreads.fastq", exact = TRUE) # with exact = FALSE by default search by name pattern a$public_file(name = "fastq") a$public_file(name = "G20479.HCC1143.2.converted.pe_1_1Mreads.fastq")
Public files are hosted in the project called admin/sbg-public-data
, and you can alternatively use the file
request to get files you need.
For public apps, there are similar API calls.
# list for 100 apps a$public_app() # list by offset and limit a$public_app(offset = 100, limit = 50) # search by id a$public_app(id = "admin/sbg-public-data/control-freec-8-1/12") # search by name in ALL apps a$public_app(name = "STAR", complete = TRUE) # search by name with exact match a$public_app(name = "Control-FREEC", exact = TRUE, complete = TRUE)
The Actions API allows you to copy files between projects in batch or send feedback to Seven Bridges.
This call allows you to copy files between projects. Unlike a$copyFile()
or file(...)$copyTo(...)
which only copy one file with each API call, this call lets you batch the copy operation and copy a list of files at a time.
p <- a$project(id = "RFranklin/source-project") f <- p$file(complete = TRUE) # get all file IDs file_ids <- sapply(f, "[[", "id") # bulk copy files to the target project req <- a$bulk_file_copy(file_ids, "RFranklin/target-project") # print the response list (req <- unname(req))
[[1]]
[[1]]$status
[1] "OK"
[[1]]$new_file_id
[1] "5bf31316e4b09b8ff39f9df7"
[[1]]$new_file_name
[1] "merged-tumor.converted.pe_1.fastq"
...
We can use sapply
to get the copy operation status, new file IDs, and (new) file names for the copied files:
sapply(req, "[[", "status")
[1] "OK" "OK" "OK" "OK"
sapply(req, "[[", "new_file_id")
[1] "5bf31316e4b09b8ff39f9df7" "5bf31316e4b09b8ff39f9dfa" "5bf31316e4b09b8ff39f9df4" "5bf31316e4b09b8ff39f9df1"
sapply(req, "[[", "new_file_name")
[1] "merged-tumor.converted.pe_1.fastq" "merged-tumor.converted.pe_2.fastq" "merged-normal.converted.pe_2.fastq"
[4] "merged-normal.converted.pe_1.fastq"
Note that a file’s ID is dependent on the project it is contained in. So, when you copy files from one project to another, their IDs will be different in the target project. The file names, however, do not change by copying, except where copying would yield non-unique file names in the target project.
Use this call to send feedbacks to Seven Bridges via the API. There are three feedback types available: "idea"
, "thought"
, or "problem"
. You can send one feedback item per minute.
a$send_feedback( "This is a test for sending feedback via API. Please ignore this message.", type = "thought" )
The Enterprise API allows you to manage your Enterprise on the Seven Bridges Platform via dedicated API calls.
Note that the authentication token for the Enterprise API is a separate token generated under the context of your division, which is different from your non-division (personal) auth token. You may see the following error when using the non-division token to call the Enterprise API:
Error: HTTP Status 400: Insufficient privileges to access this resource. Not a division user.
If you see this error, first switch to the correct division context (use the dropdown menu in the upper left corner), then click the “Developer” tab, click “Auth token”, generate the token, and use this token for API authentication.
To list all divisions you have access to, use
a <- Auth(...) a$division()
[[1]]
== Division ==
id : the-division
name : The Division
[[2]]
== Division ==
id : another-division
name : Another Division
To get the details (ID and name) of a division, for example, the first division listed above, use
(d <- a$division("the-division"))
== Division ==
id : the-division
name : The Division
Let’s create two new teams under this division:
team1 <- d$create_team(name = "New Team 1") team2 <- d$create_team(name = "New Team 2")
To check the details (ID and name) of the first team we just created, use
d$team(team1$id)
== Team ==
id : 39f6cb44-9e3c-40e7-8639-3c0f3c1e4892
name : New Team 1
We can add division members to the teams we created. For example, let’s add yourself (with the division username the-division/your_username
) and another division member (with the division username the-division/another_username
) to the first team. Also, let’s add yourself to the second team:
team1$add_member("the-division/your_username") team1$add_member("the-division/another_username") team2$add_member("the-division/your_username")
List all team members and their roles:
(m1 <- team1$member()) (m2 <- team2$member())
[[1]]
== Team Member ==
username : the-division/another_username
role : ADMIN
[[2]]
== Team Member ==
username : the-division/your_username
role : ADMIN
[[1]]
== Team Member ==
username : the-division/your_username
role : ADMIN
To rename a team, use:
team1$rename("Another Team Name")
== Team ==
id : 39f6cb44-9e3c-40e7-8639-3c0f3c1e4892
name : Another Team Name
After renaming, the team will still have the same ID, just with a different name.
The Markers API allows you to add, get, modify, and delete genetic markers on BAM files. These genetic markers can be then examined in the Seven Bridges Genome Browser.
As of November 2018, the Markers API is still an Advance Access feature. To use this API, set the advance_access
option to TRUE
:
Let’s create a new marker on a BAM file first:
# locate the project p <- a$project(id = "RFranklin/api-markers") # search for files with `.bam` in their names f <- p$file(name = ".bam") # use the first BAM file f <- p$file(id = f[[1]]$id) # create two markers m1 <- f$create_marker("The First Marker", start = 21631232, end = 21631232) m2 <- f$create_marker("The Second Marker", start = 21631156, end = 21631158, chromosome = "chr7", private = FALSE)
The possible parameters for creating the marker include
name
)start
and end
)chromosome
)private
).The marker’s name and its position are mandatory parameters, and the others are optional.
To list all markers on a file, use:
f$marker()
[[1]]
== Marker ==
id : 5bf478e5d38f185f0c23bc44
name : The First Marker
[[2]]
== Marker ==
id : 5bf478e6d38f185f0c23bc45
name : The Second Marker
We can also check the details for a specific marker:
marker_id <- m1$id f$marker(id = marker_id)
== Marker ==
id : 5bf478e5d38f185f0c23bc44
name : The First Marker
file : 5bf46c1be4b39b8f5232282e
position:
start : 21631232
end : 21631232
created_time : 2018-11-20T21:13:00Z
created_by : RFranklin
private : TRUE
We can also modify the information of existing markers:
(m1 <- m1$modify(name = "New Marker Name", end = 21631233, private = FALSE))
== Marker ==
id : 5bf478e5d38f185f0c23bc44
name : New Marker Name
file : 5bf46c1be4b39b8f5232282e
position:
start : 21631232
end : 21631233
created_time : 2018-11-20T21:13:00Z
created_by : RFranklin
private : FALSE
All of the five marker parameters mentioned above can be modified.
In the easy API, we return an object which contains the raw response from httr
as a field. You can either call response()
on that object or use the field as is.
Quick API cheat sheet (work in progress):
# 01 - Authentication ---------------------------------------------------------- getToken() # authentication methods a <- Auth(token = token) a <- Auth(token = token, platform = "cgc") a <- Auth(from = "env") a <- Auth(from = "file", profile_name = "aws-us-user") # list all API calls a$api() # API rate limit a$rate_limit() # 02 - User ------------------------------------------------------------------- a$user() a$user("RFranklin") # 03 - Billing ----------------------------------------------------------------- a$billing() a$billing(id = ..., breakdown = TRUE) a$invoice() a$invoice(id = "your_id") # 04 - Project ----------------------------------------------------------------- # create new project a$project_new(name = ..., billing_group_id = ..., description = ...) # list all project owned by you a$project() a$project(owner = "Rosalind.Franklin") # partial match p <- a$project(name = ..., id = ..., exact = TRUE) # delete p$delete() # update p$update(name = ..., description = ...) # members p$member() p$member_add(username = ...) p$member(username = ...)$update(write = ..., copy = ..., execute = ...) p$memeber(usrname = ...)$delete() # 05 - File -------------------------------------------------------------------- # list all files in this project p$file() # list all public files a$file(visibility = "public") # copy a$copyFile(c(fid, fid2), project = pid) # delete p$file(id = fid)$delete() # download p$file()[[1]]$download_url() p$file(id = fid3)$download("~/Downloads/") # download all download(p$file()) # update a file fl$update(name = ..., metadata = list(a = ..., b = ..., ...)) # metadata fl$meta() fl$setMeta() fl$setMeta(..., overwrite = TRUE) # 06 - App --------------------------------------------------------------------- a$app() # apps in a project p$app() p$app(name, id, revision = ...) a$copy_app(aid, project = pid, name = ...) # add new app p$app_add(short_name = ..., filename = ...) # 07 - Task -------------------------------------------------------------------- a$task() a$task(id = ...) a$task(status = ...) p$task() p$task(id = ...) p$task(status = ...) tsk <- p$task(id = ...) tsk$update() tsk$abort() tsk$run() tsk$download() tsk$detele() tsk$getInputs() tsk$monitor() getTaskHook() setTaskHook(status = ..., fun = ...)


Home
Dashboard
Predictions
Add a new goal
Show dynamics
Statistics
Investments
Spendings
Goals
Savings
Tania's Deposit
Profile
50%
Settings
Help
Log Out
6
John Smith
3 mins ago
Film festivals used to be do-or-die moments for movie makers. They were where...
John Smith
3 mins ago
Film festivals used to be do-or-die moments for movie makers. They were where...
John Smith
3 mins ago
Film festivals used to be do-or-die moments for movie makers. They were where...
John Smith
3 mins ago
Film festivals used to be do-or-die moments for movie makers. They were where...
See All Alerts
Dynamics
Go!
Settings 1
Settings 2
y = a
n
x
n
+ a
n-1
x
n-1
+ ... + a
2
x
2
+ a
1
x
1
+ a
0
Graphing a polynomial.
Degree of the Polynomial


Every Thursday, we feature an image from the StarPhoenix archives.
Every Thursday, we feature an image from the StarPhoenix archives. Today, we see an aerial view of Saskatoon, with the Bessborough Hotel just right of centre, from Nov. 21, 1964. (City of Saskatoon Archives StarPhoenix Collection S-SP-B5899-1)


MAKE A DIFFERENCE IN A CHILD'S LIFE TODAY
You can help young people and families affected by COVID-19 through continued support of Heart of Los Angeles.
As COVID-19 has forced the cancellation of our annual Gala, financial contributions are needed now more than ever as our families feel the economic impact of lost jobs and closed schools on top of already facing great chronic instability in their wages, housing, health, food situation, and emotional wellbeing prior to this crisis. We are eager to make sure that the pandemic does not trigger compounding life issues that could prohibit educational opportunities throughout their lives.
We thank you for helping to ensure that we, too, re-emerge ready and able to meet the even greater needs of our most vulnerable community members once the recovery phase begins.


Google Redesign
January 29, 2013
URL
- All started in 2011 when Larry Page became CEO, mandated that all major apps would be re-designed...by that summer.
- Googlers seem to think it is commonplace to carry multiple phones (ie Android and iOS). Seems to me if they preferred Android, they wouldn't carry an iOS device :)
- Previous approach was called Kanna
- Pockets of design-thinking previously, which was always subservient to speed. It really required Larry to top-down mandate design-thinking to break this mold.
- "Design is not just about making things more useful, it's not just about making things more beautiful, but it's about what is the right thing to make, and how to make it right." -- Matias Duarte, Director of User Experience, Android
- Vision was, Google could be better than instant - it could be almost psychic, and that was very provocative.
- Design is iteration, and only after you've tried everything, can you identify what you haven't tried (and do that ).
- A few years ago, there was no universal design language for approaching different platforms - they have worked actively on this.
- Design Process:lots of mock-ups and reviews and throwing stuff out


#include <BotBestMovesS.h>
Public Member Functions | |
| virtual const char * | SOLVER_NAME () |
| BotBestMovesS (Base *base, Level *level, int maxNodeNumber, int maxRamSize, int openTableSize, int closeTableSize, int costLimit, int deadlockedBoxesSearch=2) | |
| virtual int | f (TreeNode *treeNode) const |
| virtual int | h (TreeNode *treeNode) const |
| BotBestMovesS::BotBestMovesS | ( | Base * | base, | |
| Level * | level, | |||
| int | maxNodeNumber, | |||
| int | maxRamSize, | |||
| int | openTableSize, | |||
| int | closeTableSize, | |||
| int | costLimit, | |||
| int | deadlockedBoxesSearch = 2 | |||
| ) |
Constructor for a solver of a level
| base | main class of the game | |
| level | Level to be used | |
| maxNodeNumber | Limit number of nodes to explore | |
| maxRamSize | Limit max ram size to allocate | |
| openTableSize | size of Open Table to allocate (hashtable) | |
| closeTableSize | size of Close Table to allocate (hashtable) | |
| limit | of accepted f(x). if costLimit = -1, costLimit = +infinity | |
| deadlockedBoxesSearch | number of boxes to use when testing every possible positions of deadlock |
| virtual const char* BotBestMovesS::SOLVER_NAME | ( | ) | [inline, virtual] |
Name of this solver
Implements BotA.
| int BotBestMovesS::f | ( | TreeNode * | treeNode | ) | const [virtual] |
Cost function to evaluate the cost of a treenode with its position in the tree (parent of treenode must be assigned before calling this function) this cost function will compute number of moves from the start until this node. So we can search min moves number first WARNING : not finished yet, to be really efficient, h(x) must be definied
| int BotBestMovesS::h | ( | TreeNode * | treeNode | ) | const [virtual] |
Cost function to evaluate the cost of admissibly estimated distance to a goal treenode from this treenode (parent of treenode must be assigned before calling this function)
Implements BotA.


The object browser is a drilldown widget for probing the values of model variables. Since model variables are implemented as TCL commands, this tool lists all TCL commands. Commands containg a ``.'' in their name belong to objects, so these commands are grouped and labelled in blue. TCL namespaces are grouped and labelled in green. Clicking on this blue group opens another window containing just that group of commands. Clicking on a green name drills down into a namespace. Clicking on a red command runs it, and the result is displayed next to it -- if this is a Tcl_obj instance variable, the result is the value of the variable. You can specify an argument to the command in the Args input box.
Shift clicking allows you to select a range of commands to execute, and Control clicking allows a non-contiguous selection to be made. This is more useful for the Repeat option, which repeats the selected commands every second, allowing you to dynamically follow a model's variables.
Right mouse click on any item brings up a menu of options.


Simple charts can be rendered using an <svg>
element.
The following example shows a basic SVG chart within an Angular context.
While the markup can be written and manipulated directly, this example makes use of a third-party library, SVG.js, to demonstrate how the bars could be animated when the page loads.
When you need a more complex interactivity with a chart, we recommend using the third-party charting tool called NGX-Charts.


I've built the controller using the design & boards from here
http://openenergymonitor.org/emon/node/2044
and have tested the controller using the input from a 500W lamp to simulate the PV and a 3 KW kettle to simulate the load. I've run the voltage calibration and phase calibration checks and everything seems to be working fine and as I wrap extra turns around the CT to increase the simulated PV, the rate of triac firing increases.
The problem which I can't figure out is that the triac is firing with no simulated PV. IT fires even if I disconnect the CT.
I'm using the Mk2i PV Router Rev4 sketch in free running mode and the system is behaving the same in both normal and anti flicker mode.
I'm reasonably confident that the controller has been built correctly, but I'm scratching my head ( again)
Any ideas please ?
thanks
Re: Mk2 PV controller commissing problem
Are you simulating the load & P.V exactly as in the drawing here: http://openenergymonitor.org/emon/mk2/build
If you are not measuring the nett current, the control loop is not closed and the triac will continue to fire. That is the most likely explanation.
Re: Mk2 PV controller commissing problem
You could try running any of my Mk2a or Mk2i sketches in Tallymode. This will give you a good idea of whether the measurement side of the system is working correctly. You will also be able to calibrate your system accurately.
Until the measurement side is known to be working correctly, I wouldn't connect any dump load.
Re: Mk2 PV controller commissing problem
OK, tried a number of things ...
with the CT unconnected and transformer unpowered I get 2.5V on pins 1,2 & 3 of the LM358.
with the transformer powered at 240v I get 10.2v across the 150K & 15K resistors in series and 0.7v across the 15K resistor.
with the CT unconnected or shorted I am getting 0v on the voltmeter and using the rawsamples tool, the ADC values are centered at 509 ~ 511.
Voltage readings are spread nicely from 186 to 835 in the raw samples tool.
Using tallymode in mk2i rev4, I had to adjust powercal to 0.021 to get a sensible reading from a 500w light bulb as my known source. Using a 60w light bulb powercal it needed to be 0.01. I have kept a value of 0.021.
With the phasecal tool, even numbers around 3 or 4 are only giving a power factor of 0.4 for my 500w light bulb, which I would have thought was a purely resistive load (?). I have kept phasecal at 1.
I have tried running the bench test again with mk2i rev4 and I am getting the same results. I am certain I have set up the CT correctly, measuring the net current. As I increase the simulated PV load, (60 w lamp) by adding more turns, the rate of triac firing increases but triac firing continues with the pv load switched off.
thoughts again ?
thanks
Re: Mk2 PV controller commissing problem
Some of your results look fine, so there can't be too much wrong with your setup.
I have to go out now but will send you a PM about this.
Calypso
Re: Mk2 PV controller commissing problem
I ran phasecal tool with a 3kw heater and it needed a phasecal of 10 to get a power factor of 0.92.
Using the raw samples tool, a 60w light bulb as a resistive load showed a small but discernable variation in the CT output, a 500w light bulb gave a full spread wave form but the 3kw heater seemed to overload the CT, sending values from 0 to 1023 with only 1 or 2, non extreme readings.
The CT is OK as I have plugged it into emontx and it is giving good readings. Is it likely to be the LM358, too much gain ?
Re: Mk2 PV controller commissing problem
"The problem which I can't figure out is that the triac is firing with no simulated PV. IT fires even if I disconnect the CT."
You need to reverse the CT on the wire to empty the bucket, simulating export. What you are doing with the CT is not simulating PV, but home consumption and if you are importing or exporting.
Even I did the same thing till the penny dropped!
Re: Mk2 PV controller commissing problem
canary50, I'm sure you're going down the wrong route. I've just sent you a PM which will hopefully guide you in the right direction.
All of the sketches that I've posted should work equally well on either hardware platform. I've just build up an emonTx-based rig, and Mk2i_rev5 is working nicely with everything fully functional.
Re: Mk2 PV controller commissing problem
thanks Robin, fitted the 150R resistor and everything is working on the test bench as it should be. Power factor 1 on a resistive load, current and voltage traces evenly spread, phasecal now set to 1 and powercal at 0.08.
I am using the SCT 013 CT and for some reason I was under the impression it was fitted with a burden resistor so didn't fit the additional one ... but I have now.
On to the next step ....... I notice in one of your videos you have a changeover switch for solar to constant. Are you using a mains relay ?
And I might do battle again with IP addresses to get the nanode talking to emoncms ....
thanks again !
Re: Mk2 PV controller commissing problem
Yes, the CT really does need a burden resistor, otherwise there's nothing to develop a voltage across for the Arduino to measure at its ADC input.
With the standard YHDC CT, and a 150R burden, the full scale of your measurement system should be around 4 kW, which is fine for most PV systems. If you wanted to measure the consumption of an entire house, your burden would need to be correspondingly smaller. A burden of 18R sets the full-scale value at around 25 kW.
No, there's no relay, just two switches in parallel in the Live path; one manual and the other electronic. The triac doesn't seem to mind working in this way.
Re: Mk2 PV controller commissing problem
A current transformer behaves as a current source. That means (subject to its power rating) that it will generate whatever voltage is necessary to drive the current it needs to - in the case of the 100 A: 50 mA one, one 2-thousandth of the load current. If the ADC had a current input, that wouldn't be a problem. As it is, the ADC input is a voltage input and it needs the burden resistor to convert that current into a voltage.
It's quite possible that you can damage or destroy the Arduino analogue input if you leave off the burden resistor. A large c.t. when open-circuited can easily generate many thousands of volts and destroy itself by flashing over, not to mention the danger to anyone near. NEVER, ever, open-circuit a current transformer.
Re: Mk2 PV controller commissing problem
"I am using the SCT 013 CT and for some reason I was under the impression it was fitted with a burden resistor so didn't fit the additional one ... but I have now."
Good to hear you have the right numbers now, Were you using the SCT 013 000, not the SCT 013 030? (that has a built in burden),
It actually takes quite a lot to over stress a CT, if you have a 30A CT set up it may give a distorted O/P with a higher current, but in this application all it needs to do is tell what way the current is going and if there is enough surplus to fire the triac.
Re: Mk2 PV controller commissing problem
It actually takes quite a lot to over stress a CT, if you have a 30A CT set up it may give a distorted O/P with a higher current, but in this application all it needs to do is tell what way the current is going and if there is enough surplus to fire the triac.
The problem with a distorted output is that it is likely to be non-linear, getting progressively worse as the signal size increases. A Mk2-type system will only be able to balance consumption and generation accurately if everything is linear. When surplus PV is minimal or near maximum, the disparity of the energy content between mains cycles when the load is 'on' and 'off' is considerable, maybe 50:1 or greater.
With the Eco Eye monitor, the voltage sensor is indeed only used to show which way the current is going. That product assumes a standard value for the voltage so will be somewhat less accurate than any system which uses a measured value.
Re: Mk2 PV controller commissing problem
Tell me what current you'd like to put through a SCT 013 000 (up to 250 A), what burden resistor you'd like to use, and I'll give you a picture showing the waveform distortion - with the primary current wave for comparison. There's a picture of the waveform at 250 A with a 15 Ω burden here. (There's also a picture with no burden, showing a massive phase shift and equally massive distortion at just 18 A.)
Re: Mk2 PV controller commissing problem
There's a picture of the waveform at 250 A with a 15 Ω burden here.
Nice report Robert. One comment regarding:
The maximum phase error of 3º is insignificant (representing a power factor error of less than 0.0015).
That's only true for resistive loads right? i.e. cos 3 / cos 0 is pretty close to 1. But as the load gets closer to being purely reactive cos 88 / cos 85 is a long way from 1.
One of the things for which I use my system (which admittedly has a lot more dynamic range than emontx) is baseload tuning; answering questions like "Is it worth turning my microwave off at the wall?". You can see in the attached screenshot when it's just being a clock, it's almost purely reactive. An un-checked 3º phase error there would be quite significant.
Re: Mk2 PV controller commissing problem
That product assumes a standard value for the voltage so will be somewhat less accurate than any system which uses a measured value.
 My party trick is to demonstrate how my currentcost display will drop by about 30W just by turning my microwave on at the wall. Alas, it only works when the fridge is running. The currentcost counts the fridge's reactive power as real, and the microwave's capacitors help correct the fridge's inductor. Measuring current without reference to V is pretty useless in most cases.
It spawned a dodgy market in power factor correcting devices here.
Re: Mk2 PV controller commissing problem
The most power I've seen drawn from the grid as measured by the Eco Eye software was around 7.5KW, this was the washing machine, dishwasher and the wifes hair drier. Not thankfully a normal occurrence so using just I=p/v thats around the saturation point of the CT, is it a problem? I don't think it is as I have a 4KW max solar system so in any case the Mk2 would not be attempting to divert, Would I be right in assuming anything over about 17A is irrelevant as thats the max the PV will o/p? Or would it be anything over 12.5A (what the immersion draws) assuming good sunlight and not much else going on so the triac will be fully on and still exporting power to the grid?
"It actually takes quite a lot to over stress a CT, if you have a 30A CT set up" What I meant by this is that you aren't likely to to be drawing anywhere 30A at any one time, I do appreciate the importance of a good signal but as 4KW of PV or so seems to be the norm the diverter would be off. Does that sound right?
Re: Mk2 PV controller commissing problem
Under 'normal' loads with 'normal' power factors, 3° is indeed insignificant, but of course it does become significant at poor power factors (almost purely reactive loads). However, those loads tend to be very small so the overall effect still is it makes little difference. (Remember, down at the tens or hundreds of milliamps level, you're talking only a small handful of ADC counts peak-peak, so accuracy is not brilliant whichever way you look at it).
I once questioned one of the manufacturers of an "approximate" device: they admitted it multiplied current by a constant voltage and assumed unity power factor. I guess for anyone who knows no better, that's 'good enough' and they get some indication that they are saving energy. Whether they actually are or not, and the amount, is open to scrutiny.
The currentcost counts the fridge's reactive power as real, and the microwave's capacitors help correct the fridge's inductor.
 (You should do!).
Re: Mk2 PV controller commissing problem
Hmm... I don't think we're quite in the same ball-park as this device. (I think they mean 1 part in 109, not 1 part in 109 )
Re: Mk2 PV controller commissing problem
However, those loads tend to be very small so the overall effect still is it makes little difference.
Unfortunately, when you're trying to nail down your last 200W of 4am baseload, it is mostly made up of a whole lot of tiny loads, all with ugly power factors, all combined together to give a pretty nasty looking current signal.
(Remember, down at the tens or hundreds of milliamps level, you're talking only a small handful of ADC counts peak-peak, so accuracy is not brilliant whichever way you look at it).
I'm blessed with 24-bit high speed ADCs, one for each of 12 CTs and another for V, all running in parallel. A combination of analog and digital filters give me an effective bandwidth of about 2kHz and a dynamic range of about 1000:1. So if my full scale CT deflection is set for 20A say, I can accurately measure active and reactive power down to about 20mA (assuming your CT is up for it). In that environment 3° matters.
 (You should do!).
I can't say I've checked, but I will. What's the reasoning there?
Re: Mk2 PV controller commissing problem
This is turning into another picopence discussion-
"I'm blessed with 24-bit high speed ADCs, one for each of 12 CTs and another for V, all running in parallel. A combination of analog and digital filters give me an effective bandwidth of about 2kHz and a dynamic range of about 1000:1. So if my full scale CT deflection is set for 20A say, I can accurately measure active and reactive power down to about 20mA (assuming your CT is up for it). In that environment 3° matters."
What sort of industrial environment are you measuring and why?
Re: Mk2 PV controller commissing problem
What's the reasoning there?
The supply is generally inductive, so adding a capacitor will "correct" the power factor (to the suppliers benefit, not yours!) and it's heading towards resonance so the voltage will rise. You can soon see the effect if you throw a handful of components - a voltage generator, a series resistor and inductor representing the supply, and a load resistor with or without a parallel capacitor - at your favourite circuit simulation program. This is how dip compensators work.
I have but one comment about the phase errors in your case: If you have a system with that kind of resolution, you should be looking at measurement grade current transformers at the very minimum, most probably ring-core c.t's, and not cheap split-core ones. And the same goes for the voltage transformer(s). If you can't do that, then you also need to consider the variation of the phase errors with load (for the c.t.) and system voltage (for the v.t.), and possibly correct for that variation too.
Re: Mk2 PV controller commissing problem
Actually, on the V side I use a precision divider and it all sits behind isolators that are good for 2.5kV, but yes, the overall system accuracy is only as good as its weakest link. The trick is to find CTs with a nice flat amplitude error and phase error over a broad current range (and temperature range). That and use an upmarket power calibrator where you can independently control V, I and Ø (that's the bit that gets expensive).
Re: Mk2 PV controller commissing problem
The supply is generally inductive, so adding a capacitor will "correct" the power factor (to the suppliers benefit, not yours!) and it's heading towards resonance so the voltage will rise.
Not enough for me to detect with a multi-meter. Does it matter where I measure V? Isn't the rest of the house and the entire grid it's connected to, low enough impedance to make it pretty difficult for me to have any impact on V?
Re: Mk2 PV controller commissing problem
Isn't the rest of the house and the entire grid it's connected to, low enough impedance to make it pretty difficult for me to have any impact on V?
Judging by the amount of interest that a 3 kW dump-load rapidly switching on & off has aroused, I think the answer to that one has to be No!
Re: Mk2 PV controller commissing problem
Judging by the amount of interest that a 3 kW dump-load rapidly switching on & off has aroused, I think the answer to that one has to be No!
I'm not sure I see the connection. You're trying (quite successfully by the looks of your youtube videos) to match your power consumption to your power production right?
I think Robert is suggesting that a resonating LRC circuit in my house can change the voltage of the SEQ grid, but I suspect I've misunderstood what he's saying.
Re: Mk2 PV controller commissing problem
The 'test' impedance (I read that as the general worst-case) for a single phase supply for flicker is (0.24 + j 0.15)Ω in the line and (0.16 + j 0.10) Ω in the neutral at 240 V (you can of course lump the two together for this exercise). If you put those values into your simulation, add a representative resistive load, then add some parallel capacitance, you should see the voltage across the load rise. Of course, if your supply has a very high fault level, i.e. the impedance is lower than the test one, the effect will be much less than if it's marginal.
You won't of course achieve resonance - I wrote "towards resonance".
What Robin is saying is when he switches a dump load on and off, by virtue of the impedance of the supply, there is an additional voltage drop across that supply impedance and it causes the voltage you see to "dip". Repeated, frequently, it's called "flicker". What I'm saying is were that load to include sufficient parallel capacitance (and generally that's not practical but stay with the idea), then because the supply is inductively reactive, the interaction with the capacitive reactance you add can exactly cancel the voltage dip.
If you can see the power apparently reduce when you add a capacitive load, does it not mean that you aren't measuring the true real power in the first place? Sorry, but I think there is a reactive component in the quantity that you're measuring.
Re: Mk2 PV controller commissing problem
If you can see the power apparently reduce when you add a capacitive load, does it not mean that you aren't measuring the true real power in the first place? Sorry, but I think there is a reactive component in the quantity that you're measuring.
Yes, exactly. If you re-read my post where I mentioned that you'll see I was talking about my currentcost display. It has no connection to V at all and so can only measure apparent power.
My monitor has no such issues.. it even shows the fridge's reactive power on the right hand (inductive) side of the dial and the microwave's on the left hand (capacitive) side. See screenshot above for the microwave. I don't have one handy at the moment for the fridge.
Re: Mk2 PV controller commissing problem
Ah, I'd lost track and thought that you were saying that your hi-res monitor did it. I totally understand why the 'el cheapo' breeds tell lies.
Re: Mk2 PV controller commissing problem
Yes, as Robert has mentioned, I was simply pointing out that the local mains voltage will move slightly up and down as a relatively high-power load is turned off & on. This is despite being connected to the low-impedance grid. Various regulations exist which limit the amount of permitted flicker, and there has been much talk in these parts (and indeed some action too!) about this thorny topic.
I only mentioned it in reply to your claim that the grid is of a sufficiently low enough impedance to avoid there being any impact on the local mains voltage.
Re: Mk2 PV controller commissing problem
Ah, I'd lost track and thought that you were saying that your hi-res monitor did it.
That would be a worry, given its revenue-grade roots.
I totally understand why the 'el cheapo' breeds tell lies.
Yes, I only dusted off my old currentcost anecdote in response to richmc's bold claim that "in this application all it needs to do is tell what way the current is going and if there is enough surplus to fire the triac". If that's all PV-dump were doing, I doubt it would come close to nailing that spinning disk meter like it does, especially when the background load is a nice reactive mix.
I was simply pointing out that the local mains voltage will move slightly up and down as a relatively high-power load is turned off & on. This is despite being connected to the low-impedance grid.
Fair enough. Your "judging by the amount of interest" comment was too cryptic for me to connect the dots. I can imagine rapid fire switching of a 3kW load could be problematic. The load being switched in my currentcost party trick is way way smaller (~3W real, 60VAR reactive). So while the physics might dictate that a response will show up on V, in reality it's lost in the soup of Vgrid.
But speaking of grid impedance, it does show up as an issue here but typically in the other direction. Because we only get paid if we export our PV, the winning strategy here is to turn everything off except the fridge, send everyone off to work/school and run all your discretionary loads at night. On a sunny day this can result in areas of quite large production with very little consumption. Throw in a bit of grid impedance and before you know it inverters start hitting their Vmax trip-points... typically in the mid to high 260's. Voltage regulation on the grid seems to be very "chunky" and not very fast.


Class ReshapeNodeOp¶
Defined in File node_operators_unary.h
Inheritance Relationships¶
Base Type¶
public marian::UnaryNodeOp
(Struct UnaryNodeOp)
Derived Type¶
public marian::CallbackNodeOp
(Class CallbackNodeOp)
Class Documentation¶
-
class
ReshapeNodeOp
: public marian::UnaryNodeOp¶ Subclassed by marian::CallbackNodeOp
Public Functions
-
~ReshapeNodeOp
()¶
-
void
allocate
()¶
-
void
free
()¶
-
void
forward
()¶
-
void
backward
()¶
-
void
init_dependent
()¶ Initialization for backward step of top node in computation graph.
Allocates memory and sets gradient to 1 (df/df == 1).
-
void
set_zero_adjoint
()¶ Initialization for backward step of any non-top node in computation graph.
Allocates memory and sets gradient to 0 for further accumulation of gradients from all parents.
-
Tensor &
val
()¶
-
Tensor &
grad
()¶
-
virtual size_t
hash
()¶
Friends
-
friend
marian::ReshapeNodeOp::SerializationHelpers
-


Creative
Game Design
Animation
Burnt Sugar (2022)
A retelling of classic fairytale Hansel and Gretel using stock images. Created using Adobe After Effects for (Im)Possible Worlds: 2D Animation, Motion Graphics and Visual Effects.
PIE (2022)
An explainer video for the Principle of Inclusion-Exclusion in mathematics. Created using Adobe After Effects for (Im)Possible Worlds: 2D Animation, Motion Graphics and Visual Effects.
School of Computer Science (2022)
An exploration of some of the notable historical events and inventions of Carnegie Mellon University's School of Computer Science. Created using Adobe After Effects and Autodesk Maya for (Im)Possible Worlds: 2D Animation, Motion Graphics and Visual Effects.
After the Door Closes (2020)
A chair puts a book away in a bookshelf. Created using Autodesk Maya. For Introduction to 3D Animation, with attention to rigging, animation, and some sound design.
Pixel (2016)
Pixel, the Stanford OHS mascot, doing a backflip. Created using Autodesk Maya. This project was my introduction to animation, with attention to bending of the legs and arms as well as the physical mechanics of the jump itself.


Within this feedback form, I would like to obtain as much critique and advice as possible regarding the link below. Please fill out the feedback form below and click submit.
Which of the following sites did you view? (choose all that apply)
Which site would you say captured your interest the most? (choose one only)
Have you any web site development experience?
If so, how long has your experience been with web site development?
Comments & Questions


Run Once
Enable On Site
Disable Once
Disable On Site
Disable Once
Disable On Site
Run Once
Enable On Site
GlobalMobile is set to run on every page
To change this, update the
options


This is a test website. Nothing is charged.
Enroll Credential #1 with DevicePublicKey
Pay $0.01 and get devicePubKey
Login and get devicePubKey


Supplementary material for this article is available at
Relative SLR influence on extreme sea level and CF
Bivariate validation
Univariate return periods
Fig. S1. Relative SLR influence on extreme sea level and CF.
Fig. S2. Extreme values of sea level and precipitation.
Fig. S3. Comparison of the dependence between sea level and precipitation based on ERA-Interim and observation data.
Fig. S4. Comparison of the return periods of potential CF based on ERA-Interim and observation data.
Fig. S5. Probability of potential compound flood based on individual models.
Fig. S6. Changes in probability of potential compound flood driven by the astronomical tides.
Fig. S7. Changing return periods of extreme sea level (no SLR) and precipitation.
Fig. S8. Regional effects of SLR-driven changes of astronomical tide amplitudes on changing probability of potential compound flood.
Fig. S9. Procedure for computing compound flood return periods.
Fig. S10. Future change of CF return periods based on individual models.


|
mlpack
1.0.12
|
A simple linear regression algorithm using ordinary least squares. More...
Public Member Functions | |
| LinearRegression (const arma::mat &predictors, const arma::vec &responses, const double lambda=0, const bool intercept=true, const arma::vec &weights=arma::vec()) | |
| Creates the model. More... | |
| LinearRegression (const std::string &filename) | |
| Initialize the model from a file. More... | |
| LinearRegression (const LinearRegression &linearRegression) | |
| Copy constructor. More... | |
| LinearRegression () | |
| Empty constructor. More... | |
| double | ComputeError (const arma::mat &points, const arma::vec &responses) const |
| Calculate the L2 squared error on the given predictors and responses using this linear regression model. More... | |
| double | Lambda () const |
| Return the Tikhonov regularization parameter for ridge regression. More... | |
| double & | Lambda () |
| Modify the Tikhonov regularization parameter for ridge regression. More... | |
| const arma::vec & | Parameters () const |
| Return the parameters (the b vector). More... | |
| arma::vec & | Parameters () |
| Modify the parameters (the b vector). More... | |
| void | Predict (const arma::mat &points, arma::vec &predictions) const |
| Calculate y_i for each data point in points. More... | |
| std::string | ToString () const |
Private Attributes | |
| bool | intercept |
| Indicates whether first parameter is intercept. More... | |
| double | lambda |
| The Tikhonov regularization parameter for ridge regression (0 for linear regression). More... | |
| arma::vec | parameters |
| The calculated B. More... | |
A simple linear regression algorithm using ordinary least squares.
Optionally, this class can perform ridge regression, if the lambda parameter is set to a number greater than zero.
Definition at line 28 of file linear_regression.hpp.
| mlpack::regression::LinearRegression::LinearRegression | ( | const arma::mat & | predictors, |
| const arma::vec & | responses, | ||
| const double | lambda = 0 , |
||
| const bool | intercept = true , |
||
| const arma::vec & | weights = arma::vec() |
||
| ) |
Creates the model.
| predictors | X, matrix of data points to create B with. |
| responses | y, the measured data for each point in X |
| intercept | include intercept? |
| weights | observation weights |
| mlpack::regression::LinearRegression::LinearRegression | ( | const std::string & | filename | ) |
Initialize the model from a file.
| filename | the name of the file to load the model from. |
| mlpack::regression::LinearRegression::LinearRegression | ( | const LinearRegression & | linearRegression | ) |
Copy constructor.
| linearRegression | the other instance to copy parameters from. |
|
inline |
Empty constructor.
Definition at line 63 of file linear_regression.hpp.
| double mlpack::regression::LinearRegression::ComputeError | ( | const arma::mat & | points, |
| const arma::vec & | responses | ||
| ) | const |
Calculate the L2 squared error on the given predictors and responses using this linear regression model.
This calculation returns
where is the responses vector, is the matrix of predictors, and is the parameters of the trained linear regression model.
As this number decreases to 0, the linear regression fit is better.
| predictors | Matrix of predictors (X). |
| responses | Vector of responses (y). |
|
inline |
Return the Tikhonov regularization parameter for ridge regression.
Definition at line 99 of file linear_regression.hpp.
References lambda.
|
inline |
Modify the Tikhonov regularization parameter for ridge regression.
Definition at line 101 of file linear_regression.hpp.
References lambda.
|
inline |
Return the parameters (the b vector).
Definition at line 94 of file linear_regression.hpp.
References parameters.
|
inline |
Modify the parameters (the b vector).
Definition at line 96 of file linear_regression.hpp.
References parameters.
| void mlpack::regression::LinearRegression::Predict | ( | const arma::mat & | points, |
| arma::vec & | predictions | ||
| ) | const |
Calculate y_i for each data point in points.
| points | the data points to calculate with. |
| predictions | y, will contain calculated values on completion. |
| std::string mlpack::regression::LinearRegression::ToString | ( | ) | const |
|
private |
Indicates whether first parameter is intercept.
Definition at line 118 of file linear_regression.hpp.
|
private |
The Tikhonov regularization parameter for ridge regression (0 for linear regression).
Definition at line 116 of file linear_regression.hpp.
Referenced by Lambda().
|
private |
The calculated B.
Initialized and filled by constructor to hold the least squares solution.
Definition at line 111 of file linear_regression.hpp.
Referenced by Parameters().


Queen Maxima of Netherlands arrives in Pakistan
by https://www.facebook.com/dunyanewsISLAMABAD (Web Desk) - Queen Maxima of the Netherlands has arrived in Pakistan today (Monday) for a three-day visit in her capacity as the United Nations Secretary-General s Special Advocate for Inclusive Finance for Development.
According to details, officials of foreign ministry and Holland Ambassador to Pakistan welcomed the queen.
During the visit, the Queen will call on the President and the Prime Minister, besides engagements with a range of stakeholders from the public and private sectors.
It may be added that inclusive finance for development is one of the key priorities of the Government of Pakistan for which it has taken a number of steps.
Queen Maxima will also attend the launch of the Micro Payment Gateway, an initiative of the State Bank of Pakistan aimed at reducing cost of small payments and boosting digital transactions to benefit people and promote financial inclusion.
The Queen has been the UN Secretary General’s Special Advocate for Inclusive Finance for Development since 2009.
In this capacity, she is actively engaged worldwide in promoting access of individuals and enterprises to financial services at a reasonable cost, with the aim of improving economic and social development opportunities.
Queen Maxima had earlier visited Pakistan in February 2016.