[]{#1 .anchor}
AWS Pipeline
AWS Pipeline
Exported on Apr 11, 2019 5:03 PM
Recently Updated
-
{width="0.5in" height="0.5in"} (https://atrihub.atlassian.net/wiki/display/\~paravindranath)
Pradeep Ravindranath (https://atrihub.atlassian.net/wiki/display/\~paravindranath)
-
Pipeline Dictionaryupdated yesterday at 3:13 PMview change (https://atrihub.atlassian.net/wiki/pages/diffpagesbyversion.action?pageId=927367226&selectedPageVersions=27&selectedPageVersions=28)
-
{width="0.5in" height="0.5in"} (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
-
EDC Image Pipeline Plugin - APIupdated Apr 09, 2019view change (https://atrihub.atlassian.net/wiki/pages/diffpagesbyversion.action?pageId=965541910&selectedPageVersions=2&selectedPageVersions=3)
-
Image Pipeline - Parking Lotupdated Apr 08, 2019view change (https://atrihub.atlassian.net/wiki/pages/diffpagesbyversion.action?pageId=967344202&selectedPageVersions=22&selectedPageVersions=23)
-
{width="0.5in" height="0.5in"} (https://atrihub.atlassian.net/wiki/display/\~rkhemka)
Rajat Khemka (https://atrihub.atlassian.net/wiki/display/\~rkhemka)
-
Image Pipeline - ProcessDCM.pyupdated Apr 04, 2019view change (https://atrihub.atlassian.net/wiki/pages/diffpagesbyversion.action?pageId=965967879&selectedPageVersions=22&selectedPageVersions=23)
-
{width="0.5in" height="0.5in"} (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
-
Image Pipeline Components and Resourcesupdated Apr 03, 2019view change (https://atrihub.atlassian.net/wiki/pages/diffpagesbyversion.action?pageId=959873105&selectedPageVersions=4&selectedPageVersions=5)
How-to Articles
-
Page:Setup A new Image Pipeline(AWS Pipeline)
-
kb-how-to-article (https://atrihub.atlassian.net/wiki/label/AWSPIPE/kb-how-to-article)
-
runbook (https://atrihub.atlassian.net/wiki/label/AWSPIPE/runbook)
-
-
Page:How to manually trigger image pipeline in step function(AWS Pipeline)
- kb-how-to-article (https://atrihub.atlassian.net/wiki/label/AWSPIPE/kb-how-to-article)
Open issues in JIRA
Key T Created Updated Due Assignee Status Resolution
PIPE-58 (https://atrihub.atlassian.net/browse/PIPE-58?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-58?src=confmacro) Apr 08, 2019 16:51 Apr 08, 2019 16:51 Hongmei Qiu to do Unresolved PIPE-57 (https://atrihub.atlassian.net/browse/PIPE-57?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-57?src=confmacro) Apr 05, 2019 01:32 Apr 05, 2019 01:32 Hongmei Qiu to do Unresolved PIPE-56 (https://atrihub.atlassian.net/browse/PIPE-56?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-56?src=confmacro) Apr 04, 2019 23:52 Apr 04, 2019 23:52 Hongmei Qiu to do Unresolved PIPE-55 (https://atrihub.atlassian.net/browse/PIPE-55?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-55?src=confmacro) Apr 04, 2019 16:47 Apr 08, 2019 14:11 Unassigned in progress Unresolved PIPE-54 (https://atrihub.atlassian.net/browse/PIPE-54?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-54?src=confmacro) Apr 04, 2019 12:47 Apr 04, 2019 12:47 Hongmei Qiu to do Unresolved PIPE-53 (https://atrihub.atlassian.net/browse/PIPE-53?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-53?src=confmacro) Apr 03, 2019 00:45 Apr 03, 2019 00:47 Hongmei Qiu in progress Unresolved PIPE-52 (https://atrihub.atlassian.net/browse/PIPE-52?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-52?src=confmacro) Apr 03, 2019 00:45 Apr 03, 2019 00:47 Hongmei Qiu in progress Unresolved PIPE-51 (https://atrihub.atlassian.net/browse/PIPE-51?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-51?src=confmacro) Apr 02, 2019 16:28 Apr 04, 2019 16:50 Unassigned in review Unresolved PIPE-50 (https://atrihub.atlassian.net/browse/PIPE-50?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-50?src=confmacro) Apr 02, 2019 16:17 Apr 04, 2019 16:50 Unassigned in review Unresolved PIPE-49 (https://atrihub.atlassian.net/browse/PIPE-49?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-49?src=confmacro) Apr 02, 2019 16:17 Apr 04, 2019 16:50 Unassigned in review Unresolved PIPE-48 (https://atrihub.atlassian.net/browse/PIPE-48?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-48?src=confmacro) Apr 02, 2019 16:17 Apr 04, 2019 16:50 Unassigned in review Unresolved PIPE-47 (https://atrihub.atlassian.net/browse/PIPE-47?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-47?src=confmacro) Apr 02, 2019 16:17 Apr 04, 2019 16:50 Unassigned in review Unresolved PIPE-46 (https://atrihub.atlassian.net/browse/PIPE-46?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-46?src=confmacro) Apr 02, 2019 16:12 Apr 04, 2019 16:50 Unassigned in review Unresolved PIPE-45 (https://atrihub.atlassian.net/browse/PIPE-45?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-45?src=confmacro) Apr 02, 2019 16:07 Apr 04, 2019 16:50 Unassigned in review Unresolved PIPE-44 (https://atrihub.atlassian.net/browse/PIPE-44?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-44?src=confmacro) Apr 01, 2019 15:44 Apr 03, 2019 00:44 Hongmei Qiu in progress Unresolved PIPE-43 (https://atrihub.atlassian.net/browse/PIPE-43?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-43?src=confmacro) Apr 01, 2019 15:44 Apr 03, 2019 00:44 Hongmei Qiu in progress Unresolved PIPE-42 (https://atrihub.atlassian.net/browse/PIPE-42?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-42?src=confmacro) Apr 01, 2019 09:15 Apr 01, 2019 09:16 Hongmei Qiu in progress Unresolved PIPE-41 (https://atrihub.atlassian.net/browse/PIPE-41?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-41?src=confmacro) Apr 01, 2019 09:14 Apr 01, 2019 09:16 Hongmei Qiu in progress Unresolved PIPE-40 (https://atrihub.atlassian.net/browse/PIPE-40?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-40?src=confmacro) Apr 01, 2019 08:29 Apr 04, 2019 16:50 Unassigned in review Unresolved PIPE-39 (https://atrihub.atlassian.net/browse/PIPE-39?src=confmacro) {width="0.16666666666666666in" height="0.16666666666666666in"} (https://atrihub.atlassian.net/browse/PIPE-39?src=confmacro) Mar 29, 2019 16:01 Apr 04, 2019 16:50 Rajat Khemka in review Unresolved
Showing 20 out of 57 issues (https://atrihub.atlassian.net/secure/IssueNavigator.jspa?reset=true&jqlQuery=project%3DPIPE+AND+status+not+in+%28DONE%2C+RESOLVED%2C+CLOSED%29+&src=confmacro)
Meeting notes
Create meeting note
Incomplete tasks from meetings
Task report
Looking good, no incomplete tasks.
All meeting notes
Title Creator Modified
2019-04-03 - meeting notes - internal meetup on processDCM Hongmei Qiu (https://atrihub.atlassian.net/people/557058:0e04da62-9320-49f9-8df3-3a744f02137a?ref=confluence) Apr 02, 2019 2019-02-27 meeting notes - TRC Image Pipeline Hongmei Qiu (https://atrihub.atlassian.net/people/557058:0e04da62-9320-49f9-8df3-3a744f02137a?ref=confluence) Feb 27, 2019 2019-02-25 Meeting notes - TRC Image Pipeline Hongmei Qiu (https://atrihub.atlassian.net/people/557058:0e04da62-9320-49f9-8df3-3a744f02137a?ref=confluence) Feb 25, 2019 2019-02-20 Meeting notes - TRC Image Pipeline Hongmei Qiu (https://atrihub.atlassian.net/people/557058:0e04da62-9320-49f9-8df3-3a744f02137a?ref=confluence) Feb 20, 2019 2019-02-13 Meeting notes - TRC Image Pipeline Hongmei Qiu (https://atrihub.atlassian.net/people/557058:0e04da62-9320-49f9-8df3-3a744f02137a?ref=confluence) Feb 13, 2019 2019-02-11 Meeting notes - TRC Image Pipeline Hongmei Qiu (https://atrihub.atlassian.net/people/557058:0e04da62-9320-49f9-8df3-3a744f02137a?ref=confluence) Feb 11, 2019 2019-02-07 Meeting notes - TRC image pipeline Hongmei Qiu (https://atrihub.atlassian.net/people/557058:0e04da62-9320-49f9-8df3-3a744f02137a?ref=confluence) Feb 07, 2019 2019-01-30 Meeting notes - TRC Image Pipeline Hongmei Qiu (https://atrihub.atlassian.net/people/557058:0e04da62-9320-49f9-8df3-3a744f02137a?ref=confluence) Feb 05, 2019 2019-01-28 Meeting notes - TRC image pipeline Hongmei Qiu (https://atrihub.atlassian.net/people/557058:0e04da62-9320-49f9-8df3-3a744f02137a?ref=confluence) Jan 30, 2019 2019-01-25 Meeting notes - Pipelines Hongmei Qiu (https://atrihub.atlassian.net/people/557058:0e04da62-9320-49f9-8df3-3a744f02137a?ref=confluence) Jan 28, 2019
2019-01-25 Meeting notes - Pipelines
Date
25 Jan 2019
Participants
-
Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
-
Pradeep Ravindranath (https://atrihub.atlassian.net/wiki/display/\~paravindranath)
-
Stefania Bruschi (https://atrihub.atlassian.net/wiki/display/\~bruschi)
-
Gustavo Jimenez-Maggiora (https://atrihub.atlassian.net/wiki/display/\~gustavoj)
-
Emil Lerch (https://atrihub.atlassian.net/wiki/display/\~emilerch)
-
Heather Stratton
-
Adam Hunter (https://atrihub.atlassian.net/wiki/display/\~adamhunt)
Goals
-
Project Overview
-
Timeline
-
Architecture Overview
-
Challenges
-
Deliverables
-
Collaboration – discussion
-
Next Step – discussion
-
-
- Slides: https://atrihub.app.box.com/file/389348050296 (https://atrihub.app.box.com/file/389348050296)
Discussion topics
Time Item Presenter Notes
Emil suggested use Glue to retrieve and process files in chunks note for developers: look into this approach in phase 2
Emil suggested SFTP <https://aws.amazon.com/sftp/> (https://aws.amazon.com/sftp/)
vzip/ Snappy
Large file upload GUI JS components, multi-part upload
Action items
-
Hongmei Qiu > (https://atrihub.atlassian.net/wiki/display/\~hongmeiq) create IAM > readonly account for Emil on sandbox
-
Hongmei Qiu > (https://atrihub.atlassian.net/wiki/display/\~hongmeiq) share > sample phantom files with Emil
-
Hongmei Qiu > (https://atrihub.atlassian.net/wiki/display/\~hongmeiq) send out > doodle poll for weekly touch base meetings
-
Hongmei Qiu > (https://atrihub.atlassian.net/wiki/display/\~hongmeiq) create > slack space and invite Emil to it
-
Hongmei Qiu > (https://atrihub.atlassian.net/wiki/display/\~hongmeiq) share Emil > Github repo with lambda code
Decisions
2019-01-28 Meeting notes - TRC image pipeline
Date
28 Jan 2019
Participants
-
Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
-
Emil Lerch (https://atrihub.atlassian.net/wiki/display/\~emilerch)
-
Pradeep Ravindranath (https://atrihub.atlassian.net/wiki/display/\~paravindranath)
-
Tom Christensen
Goals
- Meeting recording: https://atrihub.app.box.com/folder/65237095772 (https://atrihub.app.box.com/folder/65237095772)
Discussion topics
Time Item Presenter Notes
-
AWS Batch: Job Role IAM for S3 access solved: create a role with s3 access policy → ecstatics → Tasks (<https://aws.amazon.com/blogs/compute/creating-a-simple-fetch-and-run-aws-batch-job/> (https://aws.amazon.com/blogs/compute/creating-a-simple-fetch-and-run-aws-batch-job/))
AWS Batch: environment, queue, definition - should we create them through lambda or feed them as parameters (Emil) feed them. Do not create them in lambda - no benefit, longer
What does multi-node configuration do for array jobs. For jobs that requires to run parallel execution.
What are node properties? Can we use it to by pass docker container for execution node ami is just to run ecstatics and docker container. Docker container can be used to work with node ami only at kernel level.
Cannot bypass container. for running and scaling only with AMIs use SQS → scale ec2. This is required when more control is required.
Debug RUNNABLE state is batch. Can be done ssh’ing into a single node running job instance, and checking the ecstatics agent.
ECS agent can be customized to write docker logs. Have to create our own log driver
What happens when we kill the ecs spun instance Exact relation not known. waits and runs the available/next available tasks.
VPC and subnets. Public VPC and public subnets in private VPC works.
private subnets keep the jobs in runnable state.
(emil) create an endpoint letting ec2 access to ecs - not working (have to check again recreating the ecs cluster - probably it had old configurations)
custom AMI have to include ecstatics agent.
Action items
Decisions
2019-01-30 Meeting notes - TRC Image Pipeline
Date
30 Jan 2019
Participants
-
Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
-
Pradeep Ravindranath (https://atrihub.atlassian.net/wiki/display/\~paravindranath)
-
Emil Lerch (https://atrihub.atlassian.net/wiki/display/\~emilerch)
-
Tom Christensen
Goals
- Meeting recording: https://atrihub.box.com/s/efczd3bqe0n49osxochwfwzc3aivp4s0 (https://atrihub.box.com/s/efczd3bqe0n49osxochwfwzc3aivp4s0)
Discussion topics - next meeting
-
Show status - demo
-
Best practice: where to store batch configuration variables (e.g. computing environment job queue and job definition)
-
Emil suggestion:
-
token: parameter store
-
db password: secret manager
-
for variables pass to next lambda function, e.g. computing environment, just definite those variables in each lambda function (so those info won’t be go through function call via network)
-
-
-
1 study portal per step function + 1st lambda function
-
store configurations in 1st lambda function
-
Any concerns on: passing API token through lambda to batch job as an update to job definition?
-
-
batch
-
passing parameters to batch
- override job definition is ok for same type of pipeline
-
best practice to update batch job submission
- e.g. file path, file name different for each s3 object that triggers the job run
-
batch error output to step function
-
cloudwatch logs, error text (grabs from log stream)
- what goes into cloudwatch logs? stdout and stderr …
-
aws batch return code makes its way back to step functionalt
-
alternatives: s3 log, cloudwatch insights (still needs to split stdout and stderr)
-
Emil recommended:~~mime approach, modify ecs agent to write to separate logs (won’t work for our case)~~
-
ecs environment variable: which log streams
-
error → s3 → cloudwatch log streams
-
todo: Emil share your example code with us
-
-
-
How to raise errors to next step function from batch (repeated from last question)
-
best practice - combine step 2 (process image) and step 3 (write metadata)? other options
- modify ECS agent
-
-
-
Templates/scripts: log the cost and duration matrix (Emil)
- TBD
-
Lambda CI/CD (serverless repository)
-
Emil look into seamless flow:
- TODO: share link jetbridge (through slack)
-
restrictions:
-
total amount of code (layer will count toward it)
-
space (/tmp, /opt)
-
-
-
Bastion Host: Setup Bastion Host to SSH into EC2 instances (https://atrihub.atlassian.net/wiki/spaces/APST2/pages/722272320/Setup+Bastion+Host+to+SSH+into+EC2+instances)
-
AWS batch computing environments VPC with private subnet NAT creates issue
- TBD: push this back to March
Action items
Decisions
2019-02-07 Meeting notes - TRC image pipeline
Date
07 Feb 2019
Participants
-
Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
-
Pradeep Ravindranath (https://atrihub.atlassian.net/wiki/display/\~paravindranath)
-
Emil Lerch (https://atrihub.atlassian.net/wiki/display/\~emilerch)
Goals
-
Lambda
-
lambda layer
-
Lambda CI/CD
-
serverless repository approach (TODO: Emil will look into this later)
- private
-
test lambda functions
-
step function local (they have a docker) - java application
https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local.html (https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local.html)
-
-
AWS Vault - for credentials
- https://github.com/99designs/aws-vault (https://github.com/99designs/aws-vault)
-
-
-
cost and performance
-
status:
-
cost and duration scripts (lambda, batch, etc.) (TODO: Emil will look into this later)
-
ecs agents (github fork) - stream script/container specific logs to CloudWatch - Emil: too overkill - just split the streams in our script into different s3 logs
-
mime objects - multi-part
-
s3 logs
-
TODO: Emil will provide reference links to ecs agents fork (public github) just for future reference (looking for older version)
- https://github.com/elerch/amazon-ecs-agent/tree/proxy_container (https://github.com/elerch/amazon-ecs-agent/tree/proxy_container)
-
-
-
-
Pipeline CI/CD (CloudFormation)
- bring everything together: step function, lambda, batch, docker
-
Security (move this item to next meeting)
- Batch VPC private subnet with NAT - investigation (if time permits)
-
Emil feedback on build and push script for docker
- https://github.com/atrihub/AWS-Deployment/issues/48 (https://github.com/atrihub/AWS-Deployment/issues/48)
Discussion topics
Time Item Presenter Notes
-
Action items
Decisions
2019-02-11 Meeting notes - TRC Image Pipeline
Date
11 Feb 2019
Participants
-
Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
-
Emil Lerch (https://atrihub.atlassian.net/wiki/display/\~emilerch)
-
Mihir Kavatkar (https://atrihub.atlassian.net/wiki/display/\~kavatkar)
Goals
-
file upload: aws sdk file upload
-
GUI → API → django → boto3
-
aws SDK - how to setup so the file can be uploaded directly to s3, and the hand off (key etc) is secure and light weight
-
-
can CloudTrail trigger on dynamic s3 prefix?
-
CloudTrail triggered step function, can job execution name be configured?
-
Batch job runs in separate environments?
-
Batch job runnable status
-
what caused this?
-
related to configuration?
-
-
Step function: Wait30Seconds
- stopping point
-
logs: which is easier to parse through
-
cloudwatch - metric filter (recommended)
-
s3
-
Discussion topics
Time Item Presenter Notes
-
Action items
Decisions
2019-02-13 Meeting notes - TRC Image Pipeline
Date
13 Feb 2019
Participants
-
Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
-
Emil Lerch (https://atrihub.atlassian.net/wiki/display/\~emilerch)
-
Mihir Kavatkar (https://atrihub.atlassian.net/wiki/display/\~kavatkar)
Goals
-
EDC File Upload Improvements (timeline for this pushed to Q2/Q3)
-
Runbook:
- https://atrihub.app.box.com/file/399829398942 (https://atrihub.app.box.com/file/399829398942)
-
Batch Compute Environment: switch to private subnet with NAT
- https://aws.amazon.com/security/security-bulletins/AWS-2019-002/ (https://aws.amazon.com/security/security-bulletins/AWS-2019-002/)
-
MIME multi-part generation and parsing (overkill)
- https://github.com/broadinstitute/cromwell/blob/958411830ff100b705e27e0a4f2f09c86e02c705/supportedBackends/aws/src/main/scala/cromwell/backend/impl/aws/AwsBatchJob.scala (https://github.com/broadinstitute/cromwell/blob/958411830ff100b705e27e0a4f2f09c86e02c705/supportedBackends/aws/src/main/scala/cromwell/backend/impl/aws/AwsBatchJob.scala)
curl/default/bab9ebde-71a6-4a27-8d6e-0092e39c3140\ \^\^\^ \^\^\^\^ \^\^\^\^\ | | ECS Task ID\ | ECS container name\ |\ |\ job definitioncurl -s \$ECS_CONTAINER_METADATA_URI; env{\ "DockerId": "dcc704d12b1dee27cf3d0ddc78d7a8ed6cb5bf09cc8e254d806928a71a5055e6",\ "Name": "default",\ "DockerName": "ecs-curl-1-default-a6f8d3ab8c8b8281e301",\ "Image": "appropriate/curl",\ "ImageID": "sha256:d37e1f717dc01df3a838955d29a149c569352c0991b1d7cf11b4ebca8c6c7f55",\ "Labels": {\ "com.amazonaws.ecs.cluster": "optimal_Batch_320ef3e9-690b-3be6-9009-62700d07ea28",\ "com.amazonaws.ecs.container-name": "default",\ "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-east-2:931443760666:task/bab9ebde-71a6-4a27-8d6e-0092e39c3140",\ "com.amazonaws.ecs.task-definition-family": "curl",\ "com.amazonaws.ecs.task-definition-version": "1"\ },\ "DesiredStatus": "RUNNING",\ "KnownStatus": "RUNNING",\ "Limits": {\ "CPU": 1024,\ "Memory": 128\ },\ "CreatedAt": "2019-02-13T21:34:24.353330197Z",\ "StartedAt": "2019-02-13T21:34:24.834316277Z",\ "Type": "NORMAL"\ }SHLVL=2\ HOME=/root\ AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/db4f5274-24e8-4fa4-ac51-2282a2286998\ AWS_EXECUTION_ENV=AWS_ECS_EC2\ AWS_BATCH_JOB_ID=2c517fa4-e5cd-4e8d-90b3-0fd7efcb24f5\ AWS_BATCH_JQ_NAME=optimal\ ECS_CONTAINER_METADATA_URI=http://<IP_ADDRESS>/v3/f3bf3225-88ea-4de2-a195-f285d1af1b41 (http://<IP_ADDRESS>/v3/f3bf3225-88ea-4de2-a195-f285d1af1b41)\ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\ AWS_BATCH_JOB_ATTEMPT=1\ PWD=/\ AWS_BATCH_CE_NAME=optimal
Discussion topics
Time Item Presenter Notes
-
Action items
Decisions
2019-02-20 Meeting notes - TRC Image Pipeline
Date
20 Feb 2019
Participants
-
Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
-
Emil Lerch (https://atrihub.atlassian.net/wiki/display/\~emilerch)
-
Mihir Kavatkar (https://atrihub.atlassian.net/wiki/display/\~kavatkar)
Goals
-
cost report
-
study
-
service
-
step/lambda
-
-
encryption/HIPAA
-
revisit KMS key management (re-cycle, DR)
-
Batch computing environment
- https://aws.amazon.com/about-aws/whats-new/2017/09/aws-batch-is-now-a-hipaa-eligible-service/ (https://aws.amazon.com/about-aws/whats-new/2017/09/aws-batch-is-now-a-hipaa-eligible-service/)
-
Step Function + Lambda
-
-
CloudFormation, CI/CD - best practices
-
1 per study image pipeline? or
-
1 per section?
-
walk through runbook (https://atrihub.app.box.com/file/399829398942), and identify what can be automated?
-
-
Test - step functions/lambda
-
Github → Travis CI
-
Github → CodePipeline/CodeBuild
-
Discussion topics
Time Item Presenter Notes
-
Action items
Decisions
2019-02-25 Meeting notes - TRC Image Pipeline
Attendees
-
Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
-
Mihir Kavatkar (https://atrihub.atlassian.net/wiki/display/\~kavatkar)
-
Emil Lerch (https://atrihub.atlassian.net/wiki/display/\~emilerch)
Agenda
CloudFormation
-
Step function execution name: how to customize?
-
Emil was pointing to the linux version used in the batch compute environment to test the private VPC in batch. We are currently using the managed amazon AMI. In order to do what Emil suggested I think we would need to create a custom AMI to run the docker image.
-
CloudTrail was suggested here:
- https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-cloudwatch-events-s3.html (https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-cloudwatch-events-s3.html)
-
S3 vs CloudWatch → both triggers lambda function
-
Q: when is it useful to trigger step function directly?
-
Q: S3 vs CloudWatch for logs?
-
-
Lambda trigger function → step function, example?
-
add tag to IAM role, not supported in CloudFormation?
-
solution currently used: aws cli
-
CloudFormation source tag: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-resource-tags.html (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-resource-tags.html)
-
-
An error occurred (ValidationError) when calling the UpdateStack operation: No updates are to be performed.
- how to detect that?
-
CloudFormation drift status
2019-02-27 meeting notes - TRC Image Pipeline
-
Compute Environment: use m3.medium/m5.large instead of optimal, consulted with Emil regarding our use case
-
lambda testing locally - docker container (Github)
-
without the 15 min timeout
-
load lambda through ECR - step functions (ECS cluster, docker container) → get ECS output with location to CloudWatch log → checkout outputs in the CloudWatch logs
-
2019-04-03 - meeting notes - internal meetup on processDCM
Date
02 Apr 2019
Participants
-
Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
-
Pradeep Ravindranath (https://atrihub.atlassian.net/wiki/display/\~paravindranath)
-
Rajat Khemka (https://atrihub.atlassian.net/wiki/display/\~rkhemka)
Goals
-
Travis CI
-
drop error_log api call, replace with proper print statement
-
introduce ErrorMessage class with error codes: Pipeline Dictionary
Discussion topics
Time Item Presenter Notes
-
Action items
Decisions
Product requirements
Create product requirement
Title
No content found.
Image Pipeline
Status not started
Impact low Driver Stefania Bruschi (https://atrihub.atlassian.net/wiki/display/\~bruschi), Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq) Approver Gustavo Jimenez-Maggiora (https://atrihub.atlassian.net/wiki/display/\~gustavoj) Contributors Stefania Bruschi (https://atrihub.atlassian.net/wiki/display/\~bruschi), Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq), Jia-Shing So (https://atrihub.atlassian.net/wiki/display/\~jiashins) Informed Pradeep Ravindranath (https://atrihub.atlassian.net/wiki/display/\~paravindranath) Due date 18 Jan 2019 Outcome
note: subject and participant are interchangeable, Record ID = subjecteventcrf.id, upload form name = ddcrf.name, upload field name = ddfile.name
Background
Objective
Design and create a pipeline that synchronizes image files between two s3 buckets: EDC source bucket and image bucket. Notify all stakeholders when new image files are available in the image bucket 'quarantine' folder. Track the transaction logs in the EDC database.
The pipeline should be triggered manually on a specific file or set of files.
EDC source bucket
Image file are uploaded by the EDC system users via the file uploader widget and stored in the EDC source bucket.
-
bucket name: atri-edc-trc-production-[aws account ID]
-
source folder: subjecteventcrffile
-
file path: subjecteventcrffile/[subject.id]/[subjecteventcrf.id]/[ddfile.name]/
-
file name: file version code + file extension
Image destination bucket
Image files in the destination bucket will be made available to collaborators (e.g. labs) for further process.
-
bucket name: atri-edc-trc-production-img-[aws account ID]
-
destination folder: quarantine
-
file path: quarantine/[ddcrf.name][ddfile.name]/
-
file name: [ddcrf.name]_[ddfile.name]_[participant code]_[event code]_[subjecteventcrf.id]_[subjecteventcrffile.revisionnumber]_[edcpipelinefile.id]_[timestamp] + file extension
-
note: strip out any whitespace
-
timestamp
-
Q: timestamp when file is uploaded to source bucket? destination bucket?
- when the file is first uploaded into the quarantine folder (edcpipelinefile.ts_create)
-
format: YYYYMMDD_HHMMSS
-
-
Brainstorming
Design
Flow
{width="4.677083333333333in" height="2.40625in"}
Q: should we suggest to add an unquarantine folder? to keep quarantine as read only for users? (depends on whether the SFTP wrapper can handle folder level permissions)
Track
Track the following information in the EDC database:
-
subjecteventcrffile.id
-
filepath - file path in the destination s3 bucket
-
filename - file name in the destination s3 bucket
-
FileID - unqiue ID for each image file transferred to the destination folder
- Q: can we use SubjectEventCrfFile.id as the FileID?
-
studyUID: DICOM HEADER (0020,000D) Unique identifier for the Study.
-
serieUID: DICOM HEADER (0020,000E) Unique identifier of the Series.
-
serienum: DICOM HEADER (0020,0011) A number that identifies this Series.
-
ScanID - a unique combination of studyUID and serieUID for each fileID
-
ATRIUID - Uniqeue ID store in DICOM header to identify the image set: [fileID].[scanID]
Track the following information in a log file (s3 bucket - where?):
-
execution error
-
execution result
Notification
Type of users
-
pipeline user: labs who reviews and process the image files when they are available in the quarantine folder
-
admin user: developers of the image pipeline
Type of notifications
-
for pipeline users: a digest report runs [nightly|configurable schedule]
-
create a mailing list<EMAIL_ADDRESS><EMAIL_ADDRESS>-
report includes a list of new files made available to the quarantine folder
-
report includes count of new files, count of files failed to be transferred/processed
-
-
for admin users: error logs (immediately after execution), success logs (immediately? on a schedule?)
- create a mailing list<EMAIL_ADDRESS><EMAIL_ADDRESS>-
Q: do people need to be notified when new files uploaded to the quarantine folder?
Implementation
(AWS) EDC RDS read replica
-
Create a read only replica for the EDC database, so the pipeline can find subject, crf information during the transfer
- Q: what's the delay between real database and the replica? (trigger wait time depends on this response)
Database
File information tracked in the database tables.
Pipeline.EDCPipeline
register each pipeline (e.g. pipeline for image file, pipeline for audio files, etc.
Field
ID
Code
Label
DDFiles
pipeline.EDCPipelineFile
Field Name
ID
EDCPipeline.id
SubjectEventCrfFile.id
file_name
Status Started, In Progress, Completed, Failed
has_error T/F
[]{#scroll-bookmark-93 .anchor}~~\ ~~
pipeline.EDCPipelineFileDicom
Field Name
ID EDCPipelineFile.id ATRIUID studyUID serieUID serienum number_of_instances
API
/pipeline/edcpipeline/file
input:
{
pipeline_code: xxx
subject_event_crf_file_id: xxx
*edc_pipieline_file_id: xxx (required for update)
}
output:
{
success: T/F
error_code: reference dictionary (only show if success is F)
error: xxxx (only show if success is F)
data: {"edc_pipeline_file: object}
}
/pipeline/edcpipeline/file/dicom/add_scan_info
-
post
- request body (json)
input:
{
pipeline_code: xxx,
subject_event_crf_file_id: xxx,
edc_pipieline_file_id: xxxx
scans:[
{
study_uid: xxx,
series: [
{series_uid: xxx, series_number: yyy, number_of_instance: 123},
{series_uid: blah, series_number: halb, number_of_instance: 123},
...
]
},
{
study_uid: xxx,
series: [
{series_uid: xxx, series_number: yyy, number_of_instance: 123},
{series_uid: blah, series_number: halb, number_of_instance: 123}, ...
]
}
}
output:
{
"success": true,
"error_code": reference dictionary (only show if success is F)
"error": xxxx (only show if success is F)
"data":
{
edc_pipeline_file_id: 123,
pipeline_code: xxx,
subject_event_crf_file_id: xxx,
scans: [
{
"study_uid": xxx,
"series": [
{"series_uid": aaa, "atri_uid": yyy},
{"series_uid": aaa, "atri_uid": ddd},
...
]
},
{
"studyuid": xxx,
"series": [
{"series_uid": aaa, "atri_uid": yyy},
{"series_uid": aaa, "atri_uid": ddd},
}
]
}
}
Mailing List
Study specific
-
<EMAIL_ADDRESS><EMAIL_ADDRESS>- for users (e.g. labs)
-
<EMAIL_ADDRESS><EMAIL_ADDRESS>- for admins
(AWS) S3 bucket - temporary
-
create a temporary s3 bucket used by the step function for file processing
-
bucket name: atri-edc-trc-production-img-tmp-[AWS account ID]
(AWS) Step Function
-
triggered when a new file is uploaded / updated in S3 source bucket folder "subjecteventcrffile"
-
Job name convention: [subject.id]_[subjecteventcrf.id]_[ddfile.name]_[file version code]_[timestamp]
{width="2.8678794838145234in" height="2.6041666666666665in"}
(AWS) Lambda Function
-
QUERY READ REPLICA:
-
L1 - query file info from the read replica
-
input: file path and file name from source bucket
-
output: file metadata json
-
if file is found
-
ddcrf.name
-
ddfile.name
-
participant code
-
event code
-
subjecteventcrf.id
-
subjecteventcrffile.revisionnumber
-
subjecteventcrffile.id
-
pipeline_code
-
-
if file is not found
- return ERROR
-
-
step fn - retry function, checks return status of L1 (expect raised ERROR and try for max attempt)
-
duration:15 minutes
-
frequency: every 30 sec
-
-
-
Process DCM:
-
Notes:
-
All the following functions process the same file.
-
Size may be a restriction for lambda.
-
AWS batch can an be an alternate solution.
-
-
L5 - parse file and extract the DICOM header
-
input: file path and name
-
output: DICOM header info for each scan (list)
-
studyUID
-
serieUID
-
serienum
-
-
-
L7 - write ATRIUID in DICOM header
-
L4 - copy processed file from s3 source bucket to s3 temp bucket, rename the file
-
input:
-
source file path and name
-
target file path and name
-
-
output: successful (True/False)
-
-
-
ADDITIONAL PROCESS:
- L8 - additional process (such as LONI process)
-
WRITE META DATA:
-
L6 - call API to write file metadata in study DB (set file status to "in progress")
-
input: L1 and L5 output
-
output: successful (True/False)
-
-
-
COPY TO DESTINATION:
-
L9 - copy file from S3 temp bucket to S3 destination bucket
-
input:
-
s3 temp bucket
-
s3 destination bucket
-
file path
-
file name
-
-
output: successful (True/False)
-
-
-
UPDATE STATUS:
- L10 - call API to update file status in study DB to "done"
-
ErrorSNS
-
L3 - email/log error for Admin
-
input: error
-
output: "status : error"
-
-
-
PROCESS COMPLETION SNS:
-
L11 - email / log to Admin on summary or error
-
if error: call API to update file status in study DB to "error"
-
input: status
-
output: successful (True/False)
-
-
-
USER DIGEST (This has to be post pipeline processing - separate Lambda not part of this step fn):
-
L12 - (user digest) send batch summary to users
-
input: duration (last 24 hours etc.)
-
output: successful (True/False)
-
-
(AWS) BatchEDCAWS-93
{width="5.833333333333333in" height="6.583333333333333in"}
Error Logs
Pipeline errors will be logged in the S3 bucket
-
bucket name: atri-edc-trc-production-[aws account ID]
-
source folder: image_pipelines
-
file path: image_pipelines/error_logs/[edcpipeline.code]/
-
file name: (matching closely with destination file name under quarantined folder)
- [ddcrf.name (http://ddcrf.name)]_[ddfile.name (http://ddfile.name)]_[participant code]_[event code]_[subjecteventcrf.id (http://subjecteventcrf.id)]_[subjecteventcrffile.revisionnumber]_[edcpipelinefile.id (http://edcpipelinefile.id)]_[timestamp]_error_log.txt
Error Flags in the Database
scenario 1: edcpipelinefile.status = completed, but edcpipelinefile.has_error = True
-
File is processed and is available in destination bucket quarantined folder, but some errors were generated during the process
- e.g. can't insert ATRIUID into the DICOM header etc.
scenario 2: edcpipelinefile.status = Failed
- File is not available in the destination bucket quarantined folder, something more seriously wrong during the process
Error Codes and their meaning
Please refer to Pipeline Dictionary
Q: should I promote error code to database table column? e.g. edcpipelinefileerrorlog.error_code
Error Notification Email
{width="5.9006944444444445in" height="4.06590113735783in"}
Relevant Data
ATRIUID / Batch script logic
{width="5.555555555555555in" height="4.166666666666667in"}
Brainstorming Drawing Board
{width="5.555555555555555in" height="4.166666666666667in"}
TRC Amyloid PET Scan & Data Flow
Pipeline Dictionary
Function List
code description lambda function script
L1 Lambda \[study portal name]-imgpipe-query-info
L2 Lambda imgpipe-submit-batch-job
S1 script processing image files processDMC.py
L3 Lambda imgpipe-get-batch-job-status
L4 Lambda imgpipe-copy-to-destination
L5 Lambda imgpipe-update-pipeline-status
L6 Lambda imgpipe-notification-process-error
L7 Lambda imgpipe-notification-process-completion
L8 Lambda img-pipe-create-pipeline-job
L9 Lambda img-pipe-invoke-step-function
Error List
code meaning
ERRL1001 Missing expected input
ERRL1002 Failed to create db connection
ERRL1003 EDC record not found
ERRL1004 Failed to query EDC record
ERRL2001 Error submitting Batch Job.
ERRL2002 missing required input
ERRS1001 Missing expected input parameter.
ERRS1002 Failed to read DICOM file.
ERRS1003 Return subject_event_crf_file_id / pipeline_code{} subject_event_crf_file_id / pipeline_code {} does not match
ERRS1004 Failed to call API
ERRS1005 API returned failed response.
ERRS1006 ATRIUID is empty string in the api response.
ERRS1007 ATRIUID key not present in the api response for study_uid {} and series_uid {}.
ERRS1008 ATRUID insert failed for image with study_uid {} and series_uid {}.
ERRS1009 Failed to read DICOM file during ATRIUID insert for study_uid {0} and series_uid {1}
ERRS1010 Invalid input folder location.
ERRS1011 Input files missing in input folder
ERRL3001
ERRL4001 Missing required input
ERRL4002 Error copy file to destination
ERRL4003 API call failed
ERRL8001 Failed to call API
Image Pipeline File Transfer Methods
Compare methods for external ATRI collaborators to transfer files to and from ATRI hosted s3 buckets.
CyberDuck/Mountain Duck Commander One Transfer SFTP
authentication AWS access key + AWS secret key AWS access key + AWS secret key username/password
security client-side encryption
connection HTTPS
price OpenSource/Free software
logs s3 logs s3 logs
usability client client
platform macOS, Windows mac only
setup/maintenance
Notes:
Direct connection to S3 method: s3:ListAllMyBuckets (listing all buckets or access denied error)
Image Pipeline Process Flow Diagram - High Level
{width="5.114583333333333in" height="6.78125in"}
Image Pipeline Components and Resources
EDC Plugin
Resources
-
EDC_config: https://github.com/atrihub/EDC_config (https://github.com/atrihub/EDC_config)
- EDC_IMAGE_PIPELINE_PLUGIN: flat to turn on and off plugin in EDC
-
EDC: https://github.com/atrihub/EDC (https://github.com/atrihub/EDC)
-
settings: EDC_IMAGE_PIPELINE_PLUGIN
-
authcore: image_pipeline_aws_integration group
-
account management: authentication sync tool - (note: take it out in Phase I)
-
-
edc-image-pipeline: https://github.com/atrihub/edc-plugin-image-pipeline (https://github.com/atrihub/edc-image-pipeline)
- image pipeline models, APIs etc.
-
S3 buckets:
-
atri-edc-plugins-github-deployment
-
edc-plugin-image-pipeline
-
batch
-
docker
-
lambda
-
-
-
atri-edc-logs
-
Pipeline Architecture
Resources
-
AWS-Deployment: https://github.com/atrihub/AWS-Deployment (https://github.com/atrihub/AWS-Deployment)
-
Pipeline architecture
-
Lambda function code
-
Stepfunction configuration
-
Image file processing
-
Triggers
-
CloudFormation CI/CD scripts
-
-
Runbook
Resources
Box location: https://atrihub.app.box.com/file/399829398942 (https://atrihub.app.box.com/file/399829398942)
Other Resources
Image Pipeline Architecture Diagram
{width="5.9006944444444445in" height="2.265298556430446in"}
Decision log
Create decision
Decision Status Stakeholders Outcome Due date Owner
EDC Image Pipeline Plugin - API in progress Image Pipeline - Parking Lot in progress Image Pipeline - ProcessDCM.py not started Image Pipeline - Step Function in progress Image Pipeline - Batch not started Image Pipeline - Lambda in progress AWS Consultant Suggestions Summary in progress EDC File Upload Improvements paused EDC Image Pipeline Plugin - Model in progress
Decisions
Record important project decisions and communicate them with your team.
Create from template
EDC File Upload Improvements
Add your comments directly to the page. Include links to any relevant research, data, or feedback.
Status paused
Impact medium
Driver Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
Approver
Contributors Gustavo Jimenez-Maggiora (https://atrihub.atlassian.net/wiki/display/\~gustavoj) Jia-Shing So (https://atrihub.atlassian.net/wiki/display/\~jiashins) Mihir Kavatkar (https://atrihub.atlassian.net/wiki/display/\~kavatkar)
Informed
Due date
Outcome
Background
File uploading is very slow via the EDC file attachment widget.
The issue is we are loading everything in memory when go through multiple layers to upload a file (GUI → API → Django → boto3 → write to S3 bucket).
We need to look into a more optimal solution to improve the performance of the file attachment widget.
Relevant data
-
AWS SDK for javascript
- https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/s3-example-photo-album.html (https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/s3-example-photo-album.html)
-
AWS Security Token Service STS
- https://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html (https://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html)
Options considered
Upload file directory to s3 bucket using AWS SDK for Javascript and AWS STS (for authentication).
Flow:
Client browser file uploaded → calls Django API to get a session token via sts → with the access token AWS SDK for Javascript uploads the file into s3 bucket, returns object key → calls Django API to update SubjectEventCrfFile table with the file info
use STS assumed_role() to get the session token, set the duration to 300sec (min value):
(access key, secret key, session token) = sts.assumed_role(upload_only_role_policy, s3 key, duration seconds)
- important: make sure to scoping down the role to only upload a specific object
Action items
Outcome
AWS Consultant Suggestions Summary
Add your comments directly to the page. Include links to any relevant research, data, or feedback.
Status in progress
Impact medium
Driver Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
Approver
Contributors Pradeep Ravindranath (https://atrihub.atlassian.net/wiki/display/\~paravindranath) Gustavo Jimenez-Maggiora (https://atrihub.atlassian.net/wiki/display/\~gustavoj) Stefania Bruschi (https://atrihub.atlassian.net/wiki/display/\~bruschi) Mihir Kavatkar (https://atrihub.atlassian.net/wiki/display/\~kavatkar)
Informed
Due date
Outcome
Summary
Runbook
-
IAM user "atri-img-pipe-ec2-docker", Group "atri-img-pipe-ec2-docker" (item 7)
- If we push Docker from an EC2 instance instead of from local machine, we can use the role instead a user (timeline: future)
-
NEVER grant the following policies:
-
AmazonVPCFullAccess
-
AmazonS3FullAccess
-
-
Avoid using CloudWatchLogFullAccess policy, try AWSLambdaBasicExecutionRole
-
IAM roles, one per study portal per image pipeline
-
then per lambda
-
default: atri-lambda-basic-execution-role
-
query-read-replica: atri-edc-trc-dev-img-pipe-query-read-replica-lambda-role
-
submit-batch-job: atri-edc-trc-dev-img-pipe-submit-batch-job-lambda-role
-
get-batch-job-status: atri-edc-trc-dev-img-pipe-get-batch-job-status-lambda-role
-
copy-to-destination: atri-edc-trc-dev-img-pipe-copy-to-destination-lambda-role
-
-
CloudFormation
- Anything marked as custom script in runbook under "CloudFormation" column can be hooked into CloudFormation (as a lambda function) using Custom Resources (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html)
Docker
-
When creating the temporary EC2 instance, do not use the same VPC for EDC/image pipeline (e.g. use the default VPC)
-
Use the latest AMI for docker: amazonlinux:2018.03
Trigger
- since we have the CloudWatch logs, we don't need to setup the CloudTrail
Relevant data
runbook (https://atrihub.app.box.com/file/399829398942)
Action items
Outcome
Design for unit testing
{width="5.9006944444444445in" height="1.21208552055993in"}
Flow #1\ code stored in github -> pull request -> triggers test run on Travis CI -> result + coverage reports come back to Github
reference .travis.yml configuration: https://github.com/nicor88/aws-python-lambdas/blob/master/.travis.yml (https://github.com/nicor88/aws-python-lambdas/blob/master/.travis.yml)
target: unit tests only for phase I
Lambda functions:
-
img-pipe-invoke-step-function.py
-
img-pipe-create-pipeline-job.py
-
img-pipe-query-read-replica.py
-
img-pipe-submit-batch-job.py
-
img-pipe-get-batch-job-status.py
-
img-pipe-copy-to-destination.py
-
img-pipe-notification-for-error.py
-
img-pipe-notification-for-completion.py
Image Processing script:
- processDCM.py
.travis.yml template
language: python
python: '3.6'
sudo: false
# We don't care about Travis' python versions, we install conda anyway
#env:
# global:
# - AWS_DEFAULT_REGION=eu-west-1
# - PYTHONPATH=\$TRAVIS_BUILD_DIR:\$PYTHONPATH
install:
pip install awscli
pip install boto3
# # install libs from the requirements of each single lambda
# - for i in src/*/; do pip install -r \$i"requirements.txt"; done
script:
# run tests
- pytest
#before_deploy:
# - mkdir -p dist
# # create zip for each lambda folder in src
# - for i in src/*/; do .travis/build_lambda.sh "\$i"; done
# - cp src/*.zip dist
# deploy:
# provider: s3
# access_key_id: \$AWS_ACCESS_KEY_ID
# secret_access_key: \$AWS_SECRET_ACCESS_KEY
# bucket: \$AWS_BUCKET
# region: \$AWS_BUCKET_REGION
# local_dir: dist
# upload-dir: deployments/lambdas/travis_build
# acl: private # keep them private
# skip_cleanup: true
# on:
# all_branches: true
notifications:
email: false
Code Block 1 Travis.yml
EDC Image Pipeline Plugin - Model
Add your comments directly to the page. Include links to any relevant research, data, or feedback.
Status in progress
Impact low
Driver Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
Approver
Contributors Gustavo Jimenez-Maggiora (https://atrihub.atlassian.net/wiki/display/\~gustavoj) Stefania Bruschi (https://atrihub.atlassian.net/wiki/display/\~bruschi) Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq) Pradeep Ravindranath (https://atrihub.atlassian.net/wiki/display/\~paravindranath)
Informed
Due date
Outcome
Database
File information tracked in the database tables.
Pipeline.EDCPipeline
register each pipeline (e.g. pipeline for image file, pipeline for audio files, etc.
Field
ID
Code
Label
DDFiles
pipeline.EDCPipelineFile
one pipeline execution per record, one source file can be processed more than once.
Field Name
ID
EDCPipeline.id (http://EDCPipeline.id)
SubjectEventCrfFile.id (http://SubjectEventCrfFile.id)
file_name
Status Started, In Progress, Completed, Failed
has_error T/F
pipeline.EDCPipelineFileDicom
Field Name
ID EDCPipelineFile.id (http://EDCPipelineFile.id) ATRIUID studyUID serieUID serienum number_of_instances
Action items
Outcome
EDC Image Pipeline Plugin - API
Add your comments directly to the page. Include links to any relevant research, data, or feedback.
Status in progress
Impact low
Driver Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
Approver
Contributors
Informed
Due date
Outcome
API
/pipeline/edcpipeline/file
input:
{
pipeline_code: xxx
subject_event_crf_file_id: xxx
*edc_pipieline_file_id: xxx (required for update)
}
output:
{
success: T/F
error_code: reference dictionary (only show if success is F)
error: xxxx (only show if success is F)
data: {"id": xxx, ...} // EDCPipelineFile object
}
/pipeline/edcpipeline/file/dicom/add_scan_info
-
post
- request body (json)
input:
{
pipeline_code: xxx,
subject_event_crf_file_id: xxx,
edc_pipieline_file_id: xxxx
scans:[
{
study_uid: xxx,
series: [
{series_uid: xxx, series_number: yyy, number_of_instance: 123},
{series_uid: blah, series_number: halb, number_of_instance: 123},
...
]
},
{
study_uid: xxx,
series: [
{series_uid: xxx, series_number: yyy, number_of_instance: 123},
{series_uid: blah, series_number: halb, number_of_instance: 123}, ...
]
}
}
output:
{
"success": true,
"error_code": reference dictionary (only show if success is F)
"error": xxxx (only show if success is F)
"data":
{
edc_pipeline_file_id: 123,
pipeline_code: xxx,
subject_event_crf_file_id: xxx,
scans: [
{
"study_uid": xxx,
"series": [
{"series_uid": aaa, "atri_uid": yyy},
{"series_uid": aaa, "atri_uid": ddd},
...
]
},
{
"studyuid": xxx,
"series": [
{"series_uid": aaa, "atri_uid": yyy},
{"series_uid": aaa, "atri_uid": ddd},
}
]
}
}
Action items
Outcome
Image Pipeline - Lambda
Add your comments directly to the page. Include links to any relevant research, data, or feedback.
Status in progress
Impact low
Driver Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
Approver
Contributors
Informed
Due date
Outcome
Naming convention
prefix + function name
-
prefix = atri-edc-[study portal instance]-[image pipeline code]
- e.g. atri-edc-trc-dev-img-pipe-amypet
-
function names:
-
query-read-replica
-
create-pipeline-job
-
submit-batch-job
-
get-batch-job-status
-
copy-to-destination
-
notification-for-error
-
notification-for-completion
-
Lambda Functions
-
query-read-replica
- query EDC source file meta data info from EDC RDS read replica
-
create-pipeline-job
- call image pipeline API to create a new job in the EDCPipelineFile table
-
submit-batch-job
-
create a new batch job to process the image file
-
copy source file to workspace s3 bucket
-
optimistically process file as DICOM format and insert ATRIUID into the DICOM header
-
-
get-batch-job-status
- check batch job status
-
copy-to-destination
- copy processed file from workspace to destination s3 bucket
-
notification-for-error
-
logs error message in s3 bucket
-
send out notification to specific recipients
-
-
notification-for-completion
- send out notification to specific recipients
Reference
Runbook: https://atrihub.app.box.com/file/399829398942 (https://atrihub.app.box.com/file/399829398942)
Action items
Outcome
Image Pipeline - Step Function
Add your comments directly to the page. Include links to any relevant research, data, or feedback.
Status in progress
Impact low
Driver Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
Approver
Contributors
Informed
Due date
Outcome
Step Function
The main component of the pipeline, setup one per pipeline.
Carries the pipeline process through the lambda functions.
-
triggered when a new file is uploaded / updated in S3 source bucket folder "subjecteventcrffile"
-
Job name convention: [subject.id (http://subject.id)]_[subjecteventcrf.id (http://subjecteventcrf.id)]_[ddfile.name (http://ddfile.name)]_[file version code]_[timestamp]
State Diagram
{width="4.950495406824147in" height="4.166666666666667in"}
Action items
Outcome
Image Pipeline - ProcessDCM.py
Add your comments directly to the page. Include links to any relevant research, data, or feedback.
Status not started
Impact high / medium / low
Driver Pradeep Ravindranath (https://atrihub.atlassian.net/wiki/display/\~paravindranath)
Approver Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq) Stefania Bruschi (https://atrihub.atlassian.net/wiki/display/\~bruschi)
Contributors
Informed
Due date
Outcome
Epic PIPE-1 (https://atrihub.atlassian.net/browse/PIPE-1)
Background
Relevant data
Options considered
Option 1: Option 2:
Description
Pros and cons {width="0.16666666666666666in" height="0.16666666666666666in"} {width="0.16666666666666666in" height="0.16666666666666666in"}
![\_scroll\_external/icons/forbidden-91ed7a9569c7f6084f7bfe65768d1813d768cfcd981b61d137b79eabd1f0fe11.png](media/image16.png){width="0.16666666666666666in" height="0.16666666666666666in"} ![\_scroll\_external/icons/forbidden-91ed7a9569c7f6084f7bfe65768d1813d768cfcd981b61d137b79eabd1f0fe11.png](media/image16.png){width="0.16666666666666666in" height="0.16666666666666666in"}
Estimated cost large medium
Action items
Updating the package name from pre-package-source to pre_package_source to resolve import issue{width="0.16666666666666666in" height="0.16666666666666666in"}PIPE-39 (https://atrihub.atlassian.net/browse/PIPE-39?src=confmacro) - in review \ Adding python path to travis.yml{width="0.16666666666666666in" height="0.16666666666666666in"}PIPE-40 (https://atrihub.atlassian.net/browse/PIPE-40?src=confmacro) - in review
processDCM_refactor.py{width="0.16666666666666666in" height="0.16666666666666666in"}PIPE-46 (https://atrihub.atlassian.net/browse/PIPE-46?src=confmacro) - in review
- Refactor the code
test_dcm_check.py{width="0.16666666666666666in" height="0.16666666666666666in"}PIPE-45 (https://atrihub.atlassian.net/browse/PIPE-45?src=confmacro) - in review
-
Add test case for pipeline code, extract meta and subject_event_crf_file_id
-
Add comments to test_insert_atriuid
-
fix the import statement
-
add path variable for input folder location
test_dcm_check_for_failure{width="0.16666666666666666in" height="0.16666666666666666in"}PIPE-47 (https://atrihub.atlassian.net/browse/PIPE-47?src=confmacro) - in review
- Add a test case to test failure scenario of dcm
test_for_atri_uid_in_data{width="0.16666666666666666in" height="0.16666666666666666in"}PIPE-48 (https://atrihub.atlassian.net/browse/PIPE-48?src=confmacro) - in review
test_for_atri_uid_key_not_present_in_data{width="0.16666666666666666in" height="0.16666666666666666in"}PIPE-49 (https://atrihub.atlassian.net/browse/PIPE-49?src=confmacro) - in review
test_for_non_dicom_files{width="0.16666666666666666in" height="0.16666666666666666in"}PIPE-50 (https://atrihub.atlassian.net/browse/PIPE-50?src=confmacro) - in review
test_for_error_messages{width="0.29170713035870516in" height="0.3125437445319335in"}PIPE-55 (https://atrihub.atlassian.net/browse/PIPE-55?src=confmacro) ( {width="0.29170713035870516in" height="0.3125437445319335in"} )
Add a non dicom file for test{width="0.16666666666666666in" height="0.16666666666666666in"}PIPE-51 (https://atrihub.atlassian.net/browse/PIPE-51?src=confmacro) - in review
{width="5.9006944444444445in" height="2.367302055993001in"}
Outcome
Image Pipeline - Batch
Add your comments directly to the page. Include links to any relevant research, data, or feedback.
Status not started
Impact high / medium / low
Driver Pradeep Ravindranath (https://atrihub.atlassian.net/wiki/display/\~paravindranath)
Approver
Contributors
Informed
Due date
Outcome
Background
Relevant data
Options considered
Option 1: Option 2:
Description
Pros and cons {width="0.16666666666666666in" height="0.16666666666666666in"} {width="0.16666666666666666in" height="0.16666666666666666in"}
![\_scroll\_external/icons/forbidden-91ed7a9569c7f6084f7bfe65768d1813d768cfcd981b61d137b79eabd1f0fe11.png](media/image16.png){width="0.16666666666666666in" height="0.16666666666666666in"} ![\_scroll\_external/icons/forbidden-91ed7a9569c7f6084f7bfe65768d1813d768cfcd981b61d137b79eabd1f0fe11.png](media/image16.png){width="0.16666666666666666in" height="0.16666666666666666in"}
Estimated cost large medium
Action items
Outcome
Image Pipeline - Parking Lot
Add your comments directly to the page. Include links to any relevant research, data, or feedback.
Status in progress
Impact low
Driver Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq)
Approver
Contributors
Informed
Due date
Outcome
Immediate Needs
-
relocate scripts (processDCM, lambda) from workspace bucket to a > central bucket
-
name: atri-edc-plugins-github-deployment
-
location:
-
processDCM: > /edc-plugin-image-pipeline/batch/job_def_scripts/
-
docker: /edc-plugin-image-pipeline/docker/
-
lambda functions: /edc-plugin-image-pipeline/lambda/
-
-
-
log errors in a central log bucket
-
name: atri-edc-logs
-
location: /[study]/edc-plugin-image-pipeline/
-
-
add edc user for API call with privs > to image-pipeline-aws-integration to EDC_Config
- name: image-pipeline-aws-integration-bot
-
setup google mailing lists:
-
<EMAIL_ADDRESS>-
<EMAIL_ADDRESS>-
-
rename github repo edc-image-pipeline to > edc-plugin-image-pipeline
-
rename processDCM.py to process_dicom.py
-
migrate image_pipeline folder from AWS-Deployment, > edc-plugin-image-pipeline repo
- review CloudWatch logs make sure they are all traceable, e.g. > include information such as study, pipeline, image type, file id, > timestamp etc. (this will be a continuous project through phase I > and phase II)
Phase II
-
data export API for image inventory:
-
EDCPipeline
-
EDCPipelineFile
-
EDCPipelineDicom
-
-
step function handle additional state 'status' = skip for query read replica lambda function (re-evaluate whether we need this state before implement)
-
Digest email notification (weekly, daily summary)
-
Revisit ProcessDCM logging error messages
-
currently logged in CloudWatch logs through print
-
determine whether we can utilize Emil's approach
-
-
(evaluate) RDS Read Replica (R&D required)
- if we can restrict access through policies, we may not need this replica
-
(evaluate) dispose workspace image files after execution is completed
-
process ECAT
-
update lambda query read replica to query API token info from EDC for user image-pipeline-aws-integration-bot
-
fetch_and_run.sh (docker)
-
replace arn:aws:lambda:us-east-1:898466741470:layer:psycopg2-py37:2 with our own layer in our account
Future
-
Codepipeline + Codebuild → deploy updated Lambda and processDCM code from Github
-
revisit code versioning (currently using Github for version control)
-
automate step to create RDS read replica (based on answers we found in Phase II, we may drop this)
-
GUI register new pipeline in EDC? (TBD)
-
Report error messages from s3 bucket
Action items
Outcome
How-to articles
Add how-to article
Title Creator Modified
Setup A new Image Pipeline Hongmei Qiu (https://atrihub.atlassian.net/people/557058:0e04da62-9320-49f9-8df3-3a744f02137a?ref=confluence) Mar 27, 2019 How to manually trigger image pipeline in step function Hongmei Qiu (https://atrihub.atlassian.net/people/557058:0e04da62-9320-49f9-8df3-3a744f02137a?ref=confluence) Mar 22, 2019
Setup A new Image Pipeline
Runbook
https://atrihub.app.box.com/file/399829398942 (https://atrihub.app.box.com/file/399829398942)
Step 0
(notes to Hongmei Qiu (https://atrihub.atlassian.net/wiki/display/\~hongmeiq): improve this!)
register a new pipeline with code and label info in the EDCPipeline table via Django Admin tool.
CI/CD
github repo: https://github.com/atrihub/AWS-Deployment (https://github.com/atrihub/AWS-Deployment)
run script: sh image_pipeline/cloudformation/scripts/create_image_pipeline.sh [configuration file]
Script can be run multiple times on the same configuration, if a component is already setup, the script will skip that component
Ouptput
pending...
Configuration File
location: image_pipeline/cloudformation/config
file format: plain text
file name suggestion: [study portal instance name]-[pipeline code].config
Variables
pending...
Example
######### you may change the following variables based on the study portal, image type etc.
DEBUG=false
IS_PRODUCTION=false
AWS_PROFILE='sand-informatics'
AWS_PROFILE_DR='dr'
# Pipeline code registered in EDC
EDC_PIPELINE_CODE='amypet'
# Image Type: default to use EDC pipeline code if it is short enough
IMG_TYPE=\$EDC_PIPELINE_CODE
# EDC Elastic Beanstalk and RDS instances
EDC_EB_APP='IMAGE-PIPELINE'
EDC_EB_ENV='hq-trc-dev'
RDS_READ_REPLICA=\$EDC_EB_ENV"-aurora"
EDC_HOST='hq-trc-dev.atrihub.mobi'
# tags
TAG_STUDY='trc'
TAG_ENVIRONMENT='development'
TAG_PIPELINE='image'
TAG_IMG_TYPE=\$IMG_TYPE
# VPC
VPC_NAME='sandbox'
# image process script
IMG_SCRIPT='file://image_pipeline/scripts/processDCM.py'
# Email Notifications, comma separated
<EMAIL_ADDRESS><EMAIL_ADDRESS>######### provide a good reason when updating the following variables
# Pipeline Name
PIPELINE_NAME=\$EDC_EB_ENV"-img-pipe-"\$IMG_TYPE
# S3 buckets
SOURCE_BUCKET_PREFIX=\$EDC_EB_ENV
DESTINATION_BUCKET_PREFIX=\$SOURCE_BUCKET_PREFIX"-img-"\$IMG_TYPE
WORKSPACE_BUCKET_PREFIX=\$DESTINATION_BUCKET_PREFIX"-ws"
# EC2 key pairs
EC2_KEY_DOCKER="atri-img-pipe-ec2-docker"
EC2_KEY_BATCH_DEBUG="atri-img-pipe-batch-debug"
# Container image
# ECR_REPO_NAME='atri-image-pipeline-docker-201803'
ECR_REPO_NAME="par-custom-fetch-and-run-new"
# IAM roles
ROLE_PREFIX=\$PIPELINE_NAME
LAMBDA_ROLE_SUFFIX="-lambda-role"
BATCH_SERVICE_ROLE="AWSBatchServiceRole"
BATCH_INSTANCE_PROFILE="AWSBatchInstanceProfile"
BATCH_JOB_ROLE=\$ROLE_PREFIX"-batch-job-role"
LAMBDA_BASIC_ROLE='atri-lambda-basic-execution-role'
LAMBDA_QUERY_READ_REPLICA_ROLE=\$ROLE_PREFIX"-query-read-replica"\$LAMBDA_ROLE_SUFFIX
LAMBDA_SUBMIT_BATCH_JOB_ROLE=\$ROLE_PREFIX"-submit-batch-job"\$LAMBDA_ROLE_SUFFIX
LAMBDA_GET_BATCH_JOB_STATUS_ROLE=\$ROLE_PREFIX"-get-batch-job-status"\$LAMBDA_ROLE_SUFFIX
LAMBDA_COPY_TO_DESTINATION_ROLE=\$ROLE_PREFIX"-copy-to-destination"\$LAMBDA_ROLE_SUFFIX
LAMBDA_NOTIFICATION_ROLE=\$ROLE_PREFIX"-notification"\$LAMBDA_ROLE_SUFFIX
STEP_FUNCTION_ROLE=\$ROLE_PREFIX"-step-function-role"
CLOUDWATCH_ROLE=\$EDC_EB_ENV"-img-pipe-cloudwatch-role"
# Lambda functions
LAMBDA_QUERY_READ_REPLICA=\$PIPELINE_NAME"-query-read-replica"
LAMBDA_CREATE_PIPELINE_JOB=\$PIPELINE_NAME"-create-pipeline-job"
LAMBDA_SUBMIT_BATCH_JOB=\$PIPELINE_NAME"-submit-batch-job"
LAMBDA_GET_BATCH_JOB_STATUS=\$PIPELINE_NAME"-get-batch-job-status"
LAMBDA_COPY_TO_DESTINATION=\$PIPELINE_NAME"-copy-to-destination"
LAMBDA_NOTIFICATION_FOR_ERROR=\$PIPELINE_NAME"-notification-for-error"
LAMBDA_NOTIFICATION_FOR_COMPLETION=\$PIPELINE_NAME"-notification-for-completion"
# CloudFormation stacks
IAM_ROLE_CF_STACK=\$EDC_EB_ENV"-img-pipe-"\$IMG_TYPE"-roles"
BATCH_CF_STACK=\$EDC_EB_ENV"-img-pipe-"\$IMG_TYPE"-batch"
LAMBDA_CF_STACK=\$EDC_EB_ENV"-img-pipe-"\$IMG_TYPE"-lambda"
STEP_FN_CF_STACK=\$EDC_EB_ENV"-img-pipe-"\$IMG_TYPE"-step-fn"
TRIGGER_CF_STACK=\$EDC_EB_ENV"-img-pipe-trigger"
Code Block 2 pipeline config file example
Brainstorming (archive this to a decision log)
Prerequisite
-
Github repo:
-
AWS-Deployment
-
EDC
-
EDC-Pipeline (future)
-
-
EDC study portal
-
readonly replica
-
pipeline plug-in (future)
-
-
S3 buckets:
-
EDC study portal bucket (source)
- naming convention: atri-edc-studyportal-aws_account_id
-
Image pipeline destination bucket (target)
- naming convention: atri-edc-studyportal-img-aws_account_id
-
Image pipeline workspace bucket (workspace)
- naming convention: atri-edc-studyportal-img-workspace-aws_account_id
-
1 source vs 1 target vs 1 workspace
- area to improve
-
-
Docker
-
IAM user with credential to push to EC2 and ECS instances
-
example on sandbox: user par-ec2-to-ecr with group par-aws-cli-access
-
area to improve
-
-
EC2 key pair 1 - docker
-
-
EC2 key pair 2 - debug batch computing environment
EDC Pipeline Plugin
pending...
Currently the pipeline code is in EDC github repo branch hq_pipeline
Create a CodePipeline (CodeBuild) that pushes EDC code to EDC study portal
Batch
Part 1: Docker
Download AWS-Deployment Github zip
Launch EC2 instance via AWS console
-
create new instance
- make sure only use the free tier one
-
use "Amazon Linux AMI 2018.03.0 (HVM), SSD Volume Type" (ami-0080e4c5bc078760e)
-
no other overrides, use the default values
-
(area to improve the security: VPC)
-
choose a key pair 1 - to shell into the EC2 instance
-
follow the popup instruction to shell into the instance
-
scp -i \~/.ssh/your_ssh_key path_to_AWS-Deployment_zip_file EC2_instance_string:\~/
- EC2_instance string example: ec2-user@ec2-3-92-236-127.compute-1.amazonaws.com:\~/.
-
shell in to EC2 instance, unzip AWS-Deployment Github zip file
-
setup IAM user credential to perform aws cli commands, so we can push to EC2 and ECS instances in the future
- aws configure
-
update the package:
- sudo yum update -y
-
Install Docker:
- sudo yum install docker -y
-
Start docker service
- sudo service docker start
-
Add the ec2-user to the docker group so you can execute Docker commands without using sudo.
- sudo usermod -a -G docker ec2-user
-
if error: no basic auth credentials
-
eval \$(aws ecr get-login --no-include-email | sed 's|https://||')
- return: docker login ....
-
sudo docker login ...
-
-
go to AWS-Deployment folder: /trc_image_pipeline/pre-packaged-source/docker
-
create / update docker image in ECR within EC2 instance
-
sh build_and_push.sh <ECR image name>
-
note: batch script to process the images (AWS-Deployment)
-
-
note: after dock image is created and stored to ECR, we can terminate the EC2 instance
Part 1.1: Creating an encrypted compute resource AMI
Launch EC2 instance via AWS console
-
create new instance
- make sure only use the free tier one
-
use "Amazon Linux AMI 2018.03.0 (HVM), SSD Volume Type" (ami-0080e4c5bc078760e)
-
no other overrides, use the default values
-
choose a key pair 1 - to shell into the EC2 instance
-
shell into EC2 instance
-
update the package:
- sudo yum update -y
-
Install Docker:
- sudo yum install docker -y
-
Start docker service
- sudo service docker start
-
Add the ec2-user to the docker group so you can execute Docker commands without using sudo.
- sudo usermod -a -G docker ec2-user
-
Install the ecs agent
-
sudo yum install -y ecs-init
-
(optional) sudo start ecs
-
-
Create /etc/ecs/ecs.config
- insert the following lines:\ ECS_ENABLE_TASK_IAM_ROLE=true\ ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true\ ECS_LOGFILE=/log/ecs-agent.log\ ECS_AVAILABLE_LOGGING_DRIVERS=["json-file","awslogs"]\ ECS_LOGLEVEL=info\ ECS_CLUSTER=*default\ *
-
(optional) sudo stop ecs
-
sudo rm -rf /var/lib/ecs/data/ecs_agent_data.json
-
Create and encrypt AMI:
-
Select the instance
-
choose Actions → Image → create image
-
Go to IMAGES → AMIs from the side menu
-
select the created AMI
-
Copy the AMI with encrypt option selected.
-
-
Provide the AMI id when creating the resource in AWS batch to use custom AMI after checking the “Enable user-specified Ami ID” option
-
Stop or terminate the instance.References:https://medium.com/@abbyfuller/not-containers-101-bringing-your-own-ami-or-configuring-on-the-fly-8f66ca7d7eefhttps://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.htmlhttps://docs.aws.amazon.com/batch/latest/userguide/compute_resource_AMIs.html#batch-ami-spec (https://medium.com/@abbyfuller/not-containers-101-bringing-your-own-ami-or-configuring-on-the-fly-8f66ca7d7eef)
Part 2: Batch
convention: 1 environment, 1 job queue, 1 definition per pipeline
create new compute environment
-
managed
-
service role: create new or use existing AWSBatchServiceRole
-
Instance role: create new ecsInstanceRole
-
key pair: key pair 2 for debugging purposes
-
instance type: (default value) optimal/m5
- area to improve: review this choice later
-
min vCPUs
-
0: production
-
1: testing
-
-
desired vCPUs: 2 (based on the instance type we choose)
- area to improve: review this value
-
Network:
-
testing: sanbox
-
production: edc-xxx
-
note: keep public for now, need to improve security by using private with NAT
-
-
Tags
-
name
-
study
-
environment
-
create a job queue
-
priority: 1
-
Enable job queue: check
-
compute environment: created in previous step
job definitions
-
name
-
job attempts: use default 1
-
execution timeout (seconds): 300
-
compute environment: created in previous step
-
job role: allows docker image to communicate to s3 bucket, or other AWS services in the future
-
e.g. par-imgpipe-ecstask-s3
-
area to improve
-
-
container image: ECR -> image URI
-
environment variables
-
SCRIPT_S3_IN: s3 bucket image workspace
-
OUTPUT_S3_OUT: img-pipe-submit-batch-job defines this value
-
INPUT_S3_IN: img-pipe-submit-batch-job defines this value
-
OUT_NAME: img-pipe-submit-batch-job defines this value
-
-
parameters: will be defined by img-pipe-submit-batch-job
-
pipeline_code
-
subjecteventcrffile_id
-
edc_pipeline_file_id
-
api_token
-
api_url_dicom
-
api_url_log_error
-
-
command: value will be overwritten by the Lambda Submit Job
-
default sample command for pipeline: python3 processDCM.py --pipelinecode Ref::pipeline_code --subjecteventcrffileid Ref::subjecteventcrffile_id --edcpipelinefileid Ref::edc_pipeline_file_id --apitoken Ref::api_token --apiurldicom Ref::api_url_dicom --apiurllogerror Ref::api_url_log_error
- Ref::xxxx = Ref::parameter
-
-
vCPUs: 1
-
Memory(MB): 1024
- area to improve: define this value in the lambda function Submit Job as environment variable
-
Security:
-
privileged: check
-
user: nobody
- area to improve: review this value
-
area to improve: security, ask Emil
-
Job
-
submit a job (just for testing for initial setup)
-
name
-
job definition: created in previous step
-
job queue: created in previous step
-
job type: Single
-
container properties:
-
echo job name (just for testing)
-
leave blank for production
-
-
Step function
step function name: same as the pipeline code
step function role: grant step function permission to invoke lambda functions
- par-imgpipe-sfn
Lambda
-
create the following lambda functions:
-
list:
-
img-pipe-query-read-replica
-
img-pipe-create-pipeline-job
-
img-pipe-submit-batch-job
-
img-pipe-get-batch-job-status
-
img-pipe-copy-to-destination
-
img-pipe-notification-for-error
-
img-pipe-notification-for-completion
-
area to improve: refactor the script and environment variables to be able to use for another image pipeline
-
-
service role: for step function used lambda function
-
e.g. par-imgpipe
-
area to improve
-
-
VPC:
- area to improve: make sure the VPC is tight
-
environment variables:
-
img-pipe-query-read-replica
-
edc_pipeline_api_token
-
edc_pipeline_api_url_dicom
-
edc_pipeline_api_url_log_error
-
edc_pipeline_code
-
rds_db_name
-
rds_db_password
-
rds_db_username
-
rds_host
-
workspace_bucket_name
-
workspace_file_path
-
note: add layer for psycopg2 library, python version used: 3.7
-
-
img-pipe-create-pipeline-job
-
img-pipe-submit-batch-job
-
img-pipe-get-batch-job-status
-
img-pipe-copy-to-destination
-
target_bucket_name
-
target_file_path
-
-
img-pipe-notification-for-error
-
img-pipe-notification-for-completion
-
-
area to improve:
-
code refactor
-
environment variables
-
-
State Machine Definition
{
"StartAt":"Queryreadreplica",
"States":{
"Queryreadreplica":{
"Type":"Task",
"Resource":"arn:aws:lambda:us-east-1:xxx:function:img-pipe-query-read-replica",
"Next":"Create-pipeline-job",
"Retry":[
{
"ErrorEquals":[
"Error"
],
"IntervalSeconds":1,
"BackoffRate":2.0,
"MaxAttempts":3
}
],
"Catch":[
{
"ErrorEquals":[
"States.ALL"
],
"Next":"ErrorSNS"
}
]
},
"Create-pipeline-job":{
"Type":"Task",
"Resource":"arn:aws:lambda:us-east-1:xxx:function:hq-create-pipeline-job",
"Next":"SubmitJob",
"Catch":[
{
"ErrorEquals":[
"States.ALL"
],
"Next":"ErrorSNS"
}
]
},
"SubmitJob":{
"Type":"Task",
"Resource":"arn:aws:lambda:us-east-1:xxx:function:par-imgpipe-batchSubmitJob",
"Next":"GetJobStatus",
"Catch":[
{
"ErrorEquals":[
"States.ALL"
],
"Next":"ErrorSNS"
}
]
},
"GetJobStatus":{
"Type":"Task",
"Resource":"arn:aws:lambda:us-east-1:xxx:function:par-imgpipe-batchGetJobStatus",
"Next":"CheckJobStatus",
"InputPath":"\$",
"ResultPath":"\$.status"
},
"CheckJobStatus":{
"Type":"Choice",
"Choices":[
{
"Variable":"\$.status",
"StringEquals":"FAILED",
"Next":"ErrorSNS"
},
{
"Variable":"\$.status",
"StringEquals":"SUCCEEDED",
"Next":"par-imgpipe-copy2dest"
}
],
"Default":"Wait30Seconds"
},
"Wait30Seconds":{
"Type":"Wait",
"Seconds":15,
"Next":"GetJobStatus"
},
"par-imgpipe-copy2dest":{
"Type":"Task",
"Resource":"arn:aws:lambda:us-east-1:xxx:function:par-imgpipe-copy2dest",
"Next":"par-imgpipe-updateStatus",
"Catch":[
{
"ErrorEquals":[
"States.ALL"
],
"Next":"ErrorSNS"
}
]
},
"par-imgpipe-updateStatus":{
"Type":"Task",
"Resource":"arn:aws:lambda:us-east-1:xxx:function:par-imgpipe-updateStatus",
"Next":"processCompletionSNS",
"Catch":[
{
"ErrorEquals":[
"States.ALL"
],
"Next":"ErrorSNS"
}
]
},
"ErrorSNS":{
"Type":"Task",
"Resource":"arn:aws:lambda:us-east-1:xxx:function:par-imgpipe-errorSNS",
"Next":"processCompletionSNS"
},
"processCompletionSNS":{
"Type":"Task",
"Resource":"arn:aws:lambda:us-east-1:xxx:function:par-imgpipe-processCompletionSNS",
"End":true
}
}
}
Trigger
-
we have the state machine (step function)
-
we have source s3 bucket
-
create trail in CloudTrail
-
trail name: same as step function for batch
-
s3 bucket: workspace bucket
-
prefix: subjecteventcrffile
-
write only
-
-
storage location (for logs)
-
s3 bucket: workspace
-
prefix: logs/pipeline_code/trigger
-
-
Relevant Documents
AWS Batch (https://atrihub.atlassian.net/wiki/spaces/RD/blog/2018/12/10/912261126/AWS+Batch)
https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-cloudwatch-events-s3.html (https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-cloudwatch-events-s3.html)
How to manually trigger image pipeline in step function
{
"detail":{
"requestParameters":{
"bucketName": source s3 bucket name,
"key":s3 object key
}
}
}


Adam R. Young
Owl City
I am fallen, flawed and imperfect.
Yet drenched in the grace and mercy
that is found in Jesus Christ, there is
Strength
Adam Young is a singer, songwriter and multi-instrumentalist born in 5th July, 1986. Known to be the founder of "Owl City", an American
electronica
project created in 2007 in
Owatonna,Minnesota.
Young created the project while experimenting with music in his parent's basement.Adam gained mainstream popularity with the release of
"Fireflies"
which includes the six-time platinum singles. He graduated from Owatonna Senior High School and then found work at a Coca-Cola warehouse. He began composing melodies in his head while working, which he would then record in his studio in his parent's basement, while he suffered from insonomia, a disease in which you couldn't sleep for long. Young has described himself as an intovert and he is a devout Christian.
Here is a timeline of Adam Young's releases in You-tube :
-
From "Ocean Eyes" (2009-10) :
From "All Things Bright and Beautiful" (2010–11) :
From "The Midsummer Station" (2011–12) :
From "The Midsummer Station—Acoustic EP" (2013) :
From "Ultraviolet and Mobile Orchestra" (2014–2015) :
If you want to know more then visit this wikipedia page of 'Owl City'
and this of Adam Young.


What to pack for an unforgettable KNP visit
The secret to packing is quite simple; less is more. Photo: Kruger Park Tours
by Landé WillemseYou do not even need specialist equipment. The most important to remember is to take comfortable clothes and avoid bright colours. Earthy hues are best – beige, brown and green. It can get chilly on the back of the safari vehicle, so layers are good.
Binoculars. ( Both for in your own vehicle or on the back of a game viewing vehicle) Roof prism binoculars are recommended because they are smaller, more compact, hardier and they let in more light.
Sunscreen (even if it is winter). Hats for the sun and beanies for winter mornings. This might be Africa, but the winter months are cold, even if the days are sunny. You are likely to get a sunburn even in the middle of winter.
A good pair of trainers or hiking shoes. There is no need to pack bulky, heavy duty hiking boots. You might also like to pack a pair of sandals for around the lodge or pool.
If you’re into birding, definitely download an app like Roberts Birds, or take a birding book along, so you can tick off lots of new sightings. Most guides have a solid knowledge of birds, so even if you aren’t into birding, it’s interesting when they stop and point out unique birds or behaviours.
Travel adapters.
A good book. There will be down time and possibly limited access to wifi, so bring along that book you’ve been meaning to read for years.
ALSO READ: Endangered Aloe simii in jeopardy due to planned development
Lip balm. Especially in winter, when the cold air will make them very dry.
Swimsuit. Even winter days can be surprisingly warm and almost all the lodges and camps have amazing swimming pools.
A wide brimmed hat.
Sunglasses.
Camera, charger and spare memory cards.
A flashlight.
Medical kit. This can include some pain killers, any chronic medications. insect repellent, and band aid.


What's In Stock?
Find out what you can spend your money on at GeneralHobby.com.
Total: {{filtered.length}} found on {{data.date | date:'medium'}}
| |
Product |
Price |
|
|
{{product.name}}
|
{{product.regularPrice}}
{{product.specialPrice}}
{{product.regularPrice}}
|
Total: {{filtered.length}} found on {{data.date}}
There are no products. Have you run node index.js
?


Your privacy is very important to us. Accordingly, we have developed this Policy in order for you to understand how we collect, use, communicate and disclose and make use of personal information. The following outlines our privacy policy.
Some of our advertising partners may use cookies and web beacons on our site.
These third-party ad servers or ad networks use technology to the advertisements and links that appear on this site send directly to your browsers. They automatically receive your IP address when this occurs. Other technologies ( such as cookies, JavaScript, or Web Beacons ) may also be used by the third-party ad networks to measure the effectiveness of their advertisements and / or to personalize the advertising content that you see.
This site has no access to or control over these cookies that are used by third-party advertisers.
You should consult the respective privacy policies of these third-party ad servers for more detailed information on their practices as well as for instructions about how to opt-out of certain practices. This site's privacy policy does not apply to, and we cannot control the activities of, such other advertisers or web sites.
If you wish to disable cookies, you may do so through your individual browser options. More detailed information about cookie management with specific web browsers can be found at the browsers' respective websites.
We are committed to conducting our business in accordance with these principles in order to ensure that the confidentiality of personal information is protected and maintained.


Prototyping: Work in progress...


--- layout: pingy-documentation-base title: Pingy · Documentation project-name: Pingy project-id: pingy github-buttons: true description: A minimal ping server for minecraft. doc-tab: documentation doc-subtab: getting-started ---
pingy-**-.jar
) to the folder.
pingy.jar
.
pingy-start.bat
and put the following content in it:
pingy-start.bat
file and close the window once the server is loaded. A configuration file should be generated.
pingy-start.bat
, your ping server should now be good to go.


This algorithm implements the geo-indistinguishability algorithm as outlined in [1]. Noise is generated on a 2D laplacian distribution based on the epsilon value set in the plugin settings and the protection distance set in the tool. The noise is generated and applied to each point individually within the dataset - i.e. it assumes that the location of each points is completely independent of the location of any other point within the dataset. As such spatial statistics will only be relatively accurate based on the privacy level set in the tool.
Any point layer may be used as input. Note that the calculation is done on a 2d plane in the projected units so care should be taken when looking at the projection of the incoming dataset.
An offset distance for the noise calculation. Approximately 59% of points will be within this distance of their origin.
If this parameter is checked then the offset distance will be limited to the distance of the 95% confidence interval. This does limit the protection guaranteed by the service but it guarantees there will be no large outliers in terms of point distance moved.
A dataset with all the original attributes of the original dataset with the points offset by a random amount. Note the offset of each point is independent so if the spatial autocorrelation of the data is important at the original resolution the utility may be lost using this tool.
The numerical value of the 95% confidence interval. As this is a number output this is not visible unless the tool is run from the command prompt or within a model.
One example of using this value would be to create buffers around the anonymized features to that you can supply an anonymized area hat should contain the input feature (for example, for heritage features).
[1] Andrés, M.E., Bordenabe, N.E., Chatzikokolakis, K., and Palamidessi, P. 2013. 'Geo-indistinguishability: Differential Privacy for Location-Based Systems', In the Proceedings of the 2013 ACM SIGSAC conference on Computer and Communications Security (CCS'13). New York, New York, USA: ACM Press, pp. 901-914. Online at http://arxiv.org/abs/1212.1984v3


hessianScalar
Below is a demonstration of the features of the hessianScalar function See also: gradient,|hessian|,|jacobian|,|cellEig|.
Contents
clear; close all; clc;
Plot settings
figColor='w'; figColorDef='white'; faceAlpha1=1; faceAlpha2=0.3; fontSize=10; markerSize=25; edgeWidth=1; edgeColor='k'; cMap=gray(250);
1D Hessian on vector
%A random vector M=randn(10,1) %Compute Hessian H=hessianScalar(M,[],0)
M = -0.5320 1.6821 -0.8757 -0.4838 -0.7120 -1.1742 -0.1922 -0.2741 1.5301 -0.2490 H = -2.3860 -1.6485 0.1269 0.3689 0.0890 0.3976 0.3006 -0.2188 -1.3201 -1.7916
2D Hessian on a matrix (e.g. an image)
Below an example is shown for the Hessian of an image.
Load example image
%Set main folder defaultFolder = fileparts(mfilename('fullpath')); pathName=fullfile(defaultFolder,'data','PICT'); loadName=fullfile(pathName,'leaf1.jpg'); %Load name M=mean(double(importdata(loadName)),3); %Import and conver to grayscale n=4; M=M(1:n:end,1:n:end);
Computing the Hessian of the image
voxelSize=[1 1]; cellOpt=0; H=hessianScalar(M,voxelSize,cellOpt);
In the above example the third input, cellOpt, was set to zero this leads to a m x (n^2) array whereby the colum entries define the entries in a Hessian matrix and row entries relate to elements in the input array. In this case a Hessian trace image can be computed as:
Mh_trace=reshape(H(:,1)+H(:,end),size(M)); %Compute trace for visualization
Visualize results
figuremax(figColor,figColorDef); subplot(1,2,1); hold on; title('Original image','fontSize',fontSize); imagesc(M); axis equal; axis tight; axis ij; colormap(cMap); colorbar; subplot(1,2,2); hold on; title('Hessian trace','fontSize',fontSize); imagesc(Mh_trace); axis equal; axis tight; axis ij; colormap(cMap); colorbar; drawnow;
The cell output option
Alternatively if cellOpt==1 then the output is reformed into a cell array that matches the size of the input aray. Each cell entry then contains the nxn Hessian matrix. However this requires a matrix to cell and reshape operation making this approach slower than when cellOpt==0. However after conversion to cell form each cell contains a Hessian matrix and computations on these matrices can be performed using the syntax: [B]=cellfun(@my_func,H,'UniformOutput',0); Where my_func is a suitable function for the Hessian matrices. For instance to computer a trace image one can use: [B]=cellfun(@trace,A,'UniformOutput',0); This form has also been created as a function, namely cellTrace, see below.
Computing the Hessian of the image
voxelSize=[1 1]; cellOpt=1; H=hessianScalar(M,voxelSize,cellOpt);
Each cell contains a Hessian matrix, for example:
H{1,1}
ans = -0.6667 -6.6667 -6.6667 3.0000
Compute a trace image using cellTrace function:
Mh_trace=cellTrace(H);
Computing eigenvalues using cellEig function:
[Mh_V,Mh_D]=cellEig(H); %Get first and second eigenvalues [Mh_D1]=cell2mat(cellfun(@(x) x(1,1) ,Mh_D,'UniformOutput',0)); [Mh_D2]=cell2mat(cellfun(@(x) x(2,2) ,Mh_D,'UniformOutput',0));
Visualize results
figuremax(figColor,figColorDef); subplot(1,2,1); hold on; title('Hessian 1st eigenvalue','fontSize',fontSize); imagesc(Mh_D1); axis equal; axis tight; axis ij; colormap(cMap); colorbar; subplot(1,2,2); hold on; title('Hessian 2nd eigenvalue','fontSize',fontSize); imagesc(Mh_D2); axis equal; axis tight; axis ij; colormap(cMap); colorbar; drawnow;
GIBBON www.gibboncode.org
Kevin Mattheus Moerman, gibbon.toolbox@gmail.com


public interface ReferenceService
Also provides registration and retrieval logic for POJOs where supported by appropriate plug-in instances, these methods can be used by code which is not 'reference aware' to store and retrieve value types transparently.
Resolution of collections can happen at three different levels:
ListService
resolves the collection ID to a list of
child IDs, and doesn't traverse these children if they are themselves lists.
Use the getListService()
.getList()
to call this method directly on the list service if you need this
functionality, returning a list of T2Reference
resolveIdentifier
method in this service instead resolves to a fully
realized collection of the entities which those IDs reference, and does
recursively apply this to child lists, resulting in a nested collection
structure where the leaf nodes are ReferenceSet and ErrorDocument instances
and non-leaf are IdentifiedList of Identified (the super-interface for
IdentifiedList, ReferenceSet and ErrorDocument). Use this method if you want
to access the ExternalReferenceSPI and ErrorDocument entities directly
because your code can act on a particular reference type - in general in
these cases you would also be using the augmentation system to ensure that
the required reference type was actually present in the collection structure.
renderIdentifier
to a nested structure as in
resolveIdentifier
but where the structure consists of POJOs of the specified
type and lists or either list or the leaf type. This is used when your code
is reference agnostic and just requires the values in an easy to consume
fashion. Note that because this involves pulling the entire structure into
memory it may not be suitable for large data, use with caution. This method
will, unlike resolveIdentifier
, fail if the reference contains or is an error.| Modifier and Type | Method and Description |
|---|---|
boolean |
delete(List<T2Reference> references) |
boolean |
delete(T2Reference reference) |
void |
deleteReferencesForWorkflowRun(String workflowRunId) |
ErrorDocumentService |
getErrorDocumentService()
Returns the
ErrorDocumentService this ReferenceService uses, use
this when you need functionality from that service explicitly. |
ListService |
getListService()
Returns the
ListService this ReferenceService uses, use this when
you need functionality from that service explicitly. |
ReferenceSetService |
getReferenceSetService()
Returns the
ReferenceSetService this ReferenceService uses, use
this when you need functionality from that service explicitly. |
T2Reference |
referenceFromString(String reference)
Given a string representation of a T2Reference create a new T2Reference
with the correct depth etc.
|
T2Reference |
register(Object o,
int targetDepth,
boolean useConverterSPI,
ReferenceContext context)
The top level registration method is used to register either as yet
unregistered ErrorDocuments and ReferenceSets (if these are passed in and
already have an identifier this call does nothing) and arbitrarily nested
Lists of the same.
|
Object |
renderIdentifier(T2Reference id,
Class<?> leafClass,
ReferenceContext context)
Resolve the given identifier, building a POJO structure where the
non-list items are of the desired class.
|
Identified |
resolveIdentifier(T2Reference id,
Set<Class<ExternalReferenceSPI>> ensureTypes,
ReferenceContext context)
Perform recursive identifier resolution, building a collection structure
of Identified objects, any collection elements being IdentifiedLists of
Identified subclasses.
|
void |
resolveIdentifierAsynch(T2Reference id,
Set<Class<ExternalReferenceSPI>> ensureTypes,
ReferenceContext context,
ReferenceServiceResolutionCallback callback)
As resolveIdentifier but using a callback object and returning
immediately
|
Iterator<ContextualizedT2Reference> |
traverseFrom(T2Reference source,
int desiredDepth)
Initiates a traversal of the specified t2reference, traversing to
whatever level of depth is required such that all identifiers returned
within the iterator have the specified depth.
|
@Transactional(propagation=REQUIRED, readOnly=false) Identified resolveIdentifier(T2Reference id, Set<Class<ExternalReferenceSPI>> ensureTypes, ReferenceContext context) throws ReferenceServiceException
id
- the T2Reference to resolveensureTypes
- a set of ExternalReferenceSPI classes, this is used to augment
any resolved ReferenceSet instances to ensure that each one
has at least one of the specified types. If augmentation is
not required this can be set to null.context
- the ReferenceContext to use to resolve this and any
recursively resolved identifiers ReferenceServiceException
- if any problems occur during resolution@Transactional(propagation=REQUIRED, readOnly=false) void resolveIdentifierAsynch(T2Reference id, Set<Class<ExternalReferenceSPI>> ensureTypes, ReferenceContext context, ReferenceServiceResolutionCallback callback) throws ReferenceServiceException
ReferenceServiceException
- if anything goes wrong with the setup of the resolution job.
Any exceptions during the resolution process itself are
communicated through the callback object.Object renderIdentifier(T2Reference id, Class<?> leafClass, ReferenceContext context) throws ReferenceServiceException
This method will return a collection structure mirroring that of the specified T2Reference, client code should use T2Reference.getDepth() to determine the depth of this structure; a reference with depth of 0 means that the object returned is of the specified class, one of depth 1 is a list of this class and so on.
If the T2Reference contains or is an error this method will not retrieve it, and instead throws ReferenceServiceException
id
- the T2Reference to render to a POJOleafClass
- the java class for leaves in the resulting POJO structurecontext
- a reference context, potentially used if required by the
openStream methods of ExternalReferenceSPI implementations
used as sources for the POJO construction ReferenceServiceException
- if anything fails during this processStreamToValueConverterSPI
,
ValueCarryingExternalReference
@Transactional(propagation=REQUIRED, readOnly=false) T2Reference register(Object o, int targetDepth, boolean useConverterSPI, ReferenceContext context) throws ReferenceServiceException
This method is only valid on parameters of the following type :
T2Reference
- returned immediately as itself, this is needed
because it means we can register lists of existing T2ReferenceReferenceSet
- registered if not already registered,
otherwise returns existing T2ReferenceErrorDocument
- same behaviour as ReferenceSetExternalReferenceSPI
- wrapped in ReferenceSet, registered
and ID returnedErrorDocument
with no message,
registered and ID returnedIdentifiedList
of T2Reference
and its reference returned.o
- the object to register with the reference system, must comply
with and will be interpreted as shown in the type list above.targetDepth
- the depth of the top level object supplied. This is needed
when registering empty collections and error documents,
whether as top level types or as members of a collection
within the top level type. If registering a collection this is
the collection depth, so a List of ReferenceSchemeSPI would be
depth 1. Failing to specify this correctly will result in
serious problems downstream so be careful! We can't catch all
potential problems in this method (although some errors will
be trapped).useConverterSPI
- whether to attempt to use the ValueToReferenceConvertorSPI
registry (if defined and available) to map arbitrary objects
to ExternalReferenceSPI instances on the fly. The registry of
converters is generally injected into the implementation of
this service.context
- ReferenceContext to use if required by component services,
this is most likely to be used by the object to reference
converters if engaged. ReferenceServiceException
- if the object type (or, for collections, the recursive type
of its contents) is not in the allowed list or if a problem
occurs during registration. Also thrown if attempting to use
the converter SPI without an attached registry.ValueToReferenceConverterSPI
T2Reference referenceFromString(String reference)
reference
- @Transactional(propagation=SUPPORTS, readOnly=false) boolean delete(T2Reference reference) throws ReferenceServiceException
ReferenceServiceException
@Transactional(propagation=SUPPORTS, readOnly=false) boolean delete(List<T2Reference> references) throws ReferenceServiceException
ReferenceServiceException
@Transactional(propagation=SUPPORTS, readOnly=false) void deleteReferencesForWorkflowRun(String workflowRunId) throws ReferenceServiceException
ReferenceServiceException
ErrorDocumentService getErrorDocumentService()
ErrorDocumentService
this ReferenceService uses, use
this when you need functionality from that service explicitly.ReferenceSetService getReferenceSetService()
ReferenceSetService
this ReferenceService uses, use
this when you need functionality from that service explicitly.ListService getListService()
ListService
this ReferenceService uses, use this when
you need functionality from that service explicitly.@Transactional(propagation=SUPPORTS, readOnly=true) Iterator<ContextualizedT2Reference> traverseFrom(T2Reference source, int desiredDepth)
source
- the T2Reference from which to traverse. In general this is the
root of a collection structure.desiredDepth
- the desired depth of all returned T2References, must be less
than or equal to that of the source reference.ReferenceServiceException
- if unable to create the iterator for some reason. Note that
implementations are free to lazily perform the iteration so
this method may succeed but the iterator produced can fail
when used. If the iterator fails it will do so by throwing
one of the underlying sub-service exceptions.Copyright © 2015–2016 The Apache Software Foundation. All rights reserved.


ID tag is used for special and unique element while class element is used for group having common property


Stereotype threat creates a feeling of anxiety that’s hard to overcome with affirmations. Anxiety comes from the reptilian part of the brain, and when “flight” mode is activated, it becomes hard to rationalize feelings. I think that one of the best ways to overcome the feeling is to reassess whatever induced the anxiety in the first place. For example, In my case where I was anxious about seeming too inexperienced and young, I could try and reevaluate why the receptionist was surprised to see me. Maybe she wasn’t expecting anyone to show up because it was a weekend, or maybe she was surprised by what a snappy dresser I am. I think that after attempting to reassess, it could be helpful to try and affirm values and how I actually do fit in. By thinking about how I excel at other things that aren't necessarily related to the value in question, I can maintain my sense of self.


Preparation time
overnight
Cooking time
10 to 30 mins
Serves
Serves 8
Dietary
This quick and easy gluten-free salad is packed with raw vegetables, healthy seeds and wholesome grains.
This meal provides 312 kcal, 9g protein, 31g carbohydrate (of which 10g sugars), 17g fat (of which 3g saturates), 4g fibre and 0.2g salt per portion.
Put the quinoa in a bowl and cover with approximately 700ml/1¼ pints cold water. Add a pinch of salt. Set aside to soak overnight (or for at least 8 hours).
Drain and rinse the quinoa well. Put the quinoa in a saucepan, cover with water and bring to the boil. Reduce the heat and simmer for 10-12 minutes, or until tender. Remove from the heat and set aside to cool slightly for 5 minutes. Fluff with a fork to separate the grains.
Meanwhile, for the dressing, in a large bowl, whisk together all of the dressing ingredients until well combined.
Add the tomatoes to the dressing, then stir in the herbs, spring onions and quinoa. Season with salt and pepper.
Just before serving, heat a frying pan over a medium heat. Add the sesame seeds and dry-fry for 1-2 minutes, or until toasted and golden-brown, shaking the pan occasionally to prevent the seeds from burning. Remove from the pan and set aside.
To serve, pile the tabbouleh on a platter, then sprinkle over the toasted sesame seeds and goji berries.


Electron
API Demos
Windows
Create and manage
windows
Handling window
crashes and hangs
Warrior Hero
Managing Controller(s)
Midi Reader
Game
My Library
Managing Library
Browse Songs
Menus
Customize
menus
Register keyboard
shortcuts
Native User Interface
Open
external links
or system
file manager
Notifications
with and without custom image
Use system
dialogs
Put your app in the
tray
Drag
and drop files
Communication
Communicate between the
two processes
System
Get app or user
system information
Copy and paste from the
clipboard
Launch app from
protocol handler
Media
Take a
screenshot


ByteNote v2.1.0
Show download speed while updating
Show approximate time remaining while updating
Add an option to view the release notes
Bug fixes


If you have any suggestions or concerns regarding the quality of this course, please follow this link to let us know about it anonymously.
For final evaluation of the class, please write an essay describing your experience in this class, the strenght and weaknesses, as well as suggestions for improvements. Please consider all relevant areas in your evaluation including the class materials, managements, teaching techniques, and especially methods of on-line teaching. Your essays will be read and analyzed word by word and will be triangulated with other formal and informal anonymous evaluations to ensure validity of the results.
Thank you for sharing your experience and thoughts with us!


Plugin Information |
|---|
No information for the plugin 'url-change-trigger' is available. It may have been removed from distribution. |
Deprecated
This plugin has been deprecated. Please use the https://wiki.jenkins-ci.org/display/JENKINS/URLTrigger+Plugin to trigger a build when the response content from a URL invocation changes.
Additionally, the https://wiki.jenkins-ci.org/display/JENKINS/URLTrigger+Plugin fixes all the issues of this plugin.
The URL Change Trigger plugin allows you to trigger a Hudson build when the content of a URL changes.
Setting up the URL Change Trigger is easy. Once you've installed the plugin, you'll see a new entry on the list of "Build Triggers" on your project configuration page, labeled "Build when an URL's content changes." Check it and fill in the URL. (It may be a "file:" URL, if you like; that will trigger a build whenever a file on your file system changes. This can be useful if you've mounted a remote network share using NFS or SMB/CIFS.)
Hudson will download the content at that URL once a minute. If the content changes, your build will launch.
We use the URL Change Trigger at Redfin to run our automated UI tests whenever our internal staging website is updated. Redfin dev blog
Change Log
Version 1.3 (Nov 19, 2011)
- JENKINS-11810
Version 1.2 (Dec 29, 2009)
- Fix help link
- Add cause information in builds
- Updated uses of deprecated APIs


| In what year were you born? | |
| What is your gender? |
|
| What is your native language? | |
| What is your highest level of education? |
|
| What is your college-level background in linguistics, logic, and philosophy? | |
| Enter your Mechanical Turk Worker ID to validate your participation: | |
After submitting, please wait till the results are sent to the server. Your participation will not be validated otherwise!