Actions, resources, and condition keys for Amazon Simple Email Service v2
Amazon Simple Email Service v2 (service prefix:
ses
) provides the following service-specific resources, actions, and condition context keys for use in IAM permission policies.
References:
-
Learn how to configure this service .
-
View a list of the API operations available for this service .
-
Learn how to secure this service and its resources by using IAM permission policies.
Topics
Actions defined by Amazon Simple Email Service v2
You can specify the following actions in the
Action
element of an IAM policy statement. Use policies to grant permissions to perform an operation in AWS. When you use an action in a policy, you usually allow or deny access to the API operation or CLI command with the same name. However, in some cases, a single action controls access to more than one operation. Alternatively, some operations require several different actions.
The
Resource types
column of the Actions table indicates whether each action supports resource-level permissions. If there is no value for this column, you must specify all resources ("*") to which the policy applies in the
Resource
element of your policy statement. If the column includes a resource type, then you can specify an ARN of that type in a statement with that action. If the action has one or more required resources, the caller must have permission to use the action with those resources. Required resources are indicated in the table with an asterisk (*). If you limit resource access with the
Resource
element in an IAM policy, you must include an ARN or pattern for each required resource type. Some actions support multiple resource types. If the resource type is optional (not indicated as required), then you can choose to use one of the optional resource types.
The
Condition keys
column of the Actions table includes keys that you can specify in a policy statement's
Condition
element. For more information on the condition keys that are associated with resources for the service, see the
Condition keys
column of the Resource types table.
For details about the columns in the following table, see Actions table .
| Actions | Description | Access level | Resource types (*required) | Condition keys | Dependent actions |
|---|---|---|---|---|---|
| BatchGetMetricData | Grants permission to get metric data on your activity | Read | |||
| CreateConfigurationSet | Grants permission to create a new configuration set | Write | |||
| CreateConfigurationSetEventDestination | Grants permission to create a configuration set event destination | Write | |||
| CreateContact | Grants permission to create a contact | Write | |||
| CreateContactList | Grants permission to create a contact list | Write | |||
| CreateCustomVerificationEmailTemplate | Grants permission to create a new custom verification email template | Write | |||
| CreateDedicatedIpPool | Grants permission to create a new pool of dedicated IP addresses | Write | |||
| CreateDeliverabilityTestReport | Grants permission to create a new predictive inbox placement test | Write | |||
| CreateEmailIdentity | Grants permission to start the process of verifying an email identity | Write | |||
| CreateEmailIdentityPolicy | Grants permission to create the specified sending authorization policy for the given identity | Permissions management | |||
| CreateEmailTemplate | Grants permission to create an email template | Write | |||
| CreateImportJob | Grants permission to creates an import job for a data destination | Write | |||
| DeleteConfigurationSet | Grants permission to delete an existing configuration set | Write | |||
| DeleteConfigurationSetEventDestination | Grants permission to delete an event destination | Write | |||
| DeleteContact | Grants permission to delete a contact from a contact list | Write | |||
| DeleteContactList | Grants permission to delete a contact list with all of its contacts | Write | |||
| DeleteCustomVerificationEmailTemplate | Grants permission to delete an existing custom verification email template | Write | |||
| DeleteDedicatedIpPool | Grants permission to delete a dedicated IP pool | Write | |||
| DeleteEmailIdentity | Grants permission to delete an email identity | Write | |||
| DeleteEmailIdentityPolicy | Grants permission to delete the specified sending authorization policy for the given identity (an email address or a domain) | Permissions management | |||
| DeleteEmailTemplate | Grants permission to delete an email template | Write | |||
| DeleteSuppressedDestination | Grants permission to remove an email address from the suppression list for your account | Write | |||
| GetAccount | Grants permission to get information about the email-sending status and capabilities for your account | Read | |||
| GetBlacklistReports | Grants permission to retrieve a list of the deny lists on which your dedicated IP addresses or tracked domains appear | Read | |||
| GetConfigurationSet | Grants permission to get information about an existing configuration set | Read | |||
| GetConfigurationSetEventDestinations | Grants permission to retrieve a list of event destinations that are associated with a configuration set | Read | |||
| GetContact | Grants permission to return a contact from a contact list | Read | |||
| GetContactList | Grants permission to return contact list metadata | Read | |||
| GetCustomVerificationEmailTemplate | Grants permission to return the custom email verification template for the template name you specify | Read | |||
| GetDedicatedIp | Grants permission to get information about a dedicated IP address | Read | |||
| GetDedicatedIpPool | Grants permission to get information about a dedicated IP pool | Read | |||
| GetDedicatedIps | Grants permission to list the dedicated IP addresses a dedicated IP pool | Read | |||
| GetDeliverabilityDashboardOptions | Grants permission to get the status of the Deliverability dashboard | Read | |||
| GetDeliverabilityTestReport | Grants permission to retrieve the results of a predictive inbox placement test | Read | |||
| GetDomainDeliverabilityCampaign | Grants permission to retrieve all the deliverability data for a specific campaign | Read | |||
| GetDomainStatisticsReport | Grants permission to retrieve inbox placement and engagement rates for the domains that you use to send email | Read | |||
| GetEmailIdentity | Grants permission to get information about a specific identity | Read | |||
| GetEmailIdentityPolicies | Grants permission to return the requested sending authorization policies for the given identity (an email address or a domain) | Read | |||
| GetEmailTemplate | Grants permission to return the template object, which includes the subject line, HTML part, and text part for the template you specify | Read | |||
| GetImportJob | Grants permission to provide information about an import job | Read | |||
| GetSuppressedDestination | Grants permission to retrieve information about a specific email address that's on the suppression list for your account | Read | |||
| ListConfigurationSets | Grants permission to list all of the configuration sets for your account | List | |||
| ListContactLists | Grants permission to list all of the contact lists available for your account | List | |||
| ListContacts | Grants permission to list the contacts present in a specific contact list | List | |||
| ListCustomVerificationEmailTemplates | Grants permission to list all of the existing custom verification email templates for your account | List | |||
| ListDedicatedIpPools | Grants permission to list all of the dedicated IP pools for your account | List | |||
| ListDeliverabilityTestReports | Grants permission to retrieve the list of the predictive inbox placement tests that you've performed, regardless of their statuses, for your account | List | |||
| ListDomainDeliverabilityCampaigns | Grants permission to list deliverability data for campaigns that used a specific domain to send email during a specified time range | Read | |||
| ListEmailIdentities | Grants permission to list the email identities for your account | List | |||
| ListEmailTemplates | Grants permission to list all of the email templates for your account | List | |||
| ListImportJobs | Grants permission to list all of the import jobs for your account | List | |||
| ListRecommendations | Grants permission to list recommendations for your account | Read | |||
| ListSuppressedDestinations | Grants permission to list email addresses that are on the suppression list for your account | Read | |||
| ListTagsForResource | Grants permission to retrieve a list of the tags (keys and values) that are associated with a specific resource for your account | Read | |||
| PutAccountDedicatedIpWarmupAttributes | Grants permission to enable or disable the automatic warm-up feature for dedicated IP addresses | Write | |||
| PutAccountDetails | Grants permission to update your account details | Write | |||
| PutAccountSendingAttributes | Grants permission to enable or disable the ability to send email for your account | Write | |||
| PutAccountSuppressionAttributes | Grants permission to change the settings for the account-level suppression list | Write | |||
| PutAccountVdmAttributes | Grants permission to change the settings for VDM for your account | Write | |||
| PutConfigurationSetDeliveryOptions | Grants permission to associate a configuration set with a dedicated IP pool | Write | |||
| PutConfigurationSetReputationOptions | Grants permission to enable or disable collection of reputation metrics for emails that you send using a particular configuration set | Write | |||
| PutConfigurationSetSendingOptions | Grants permission to enable or disable email sending for messages that use a particular configuration set | Write | |||
| PutConfigurationSetSuppressionOptions | Grants permission to specify the account suppression list preferences for a particular configuration set | Write | |||
| PutConfigurationSetTrackingOptions | Grants permission to specify a custom domain to use for open and click tracking elements in email that you send for a particular configuration set | Write | |||
| PutConfigurationSetVdmOptions | Grants permission to override account-level VDM settings for a particular configuration set | Write | |||
| PutDedicatedIpInPool | Grants permission to move a dedicated IP address to an existing dedicated IP pool | Write | |||
| PutDedicatedIpPoolScalingAttributes | Grants permission to transition a dedicated IP pool from Standard to Managed | Write | |||
| PutDedicatedIpWarmupAttributes | Grants permission to put Dedicated IP warm up attributes | Write | |||
| PutDeliverabilityDashboardOption | Grants permission to enable or disable the Deliverability dashboard | Write | |||
| PutEmailIdentityConfigurationSetAttributes | Grants permission to associate a configuration set with an email identity | Write | |||
| PutEmailIdentityDkimAttributes | Grants permission to enable or disable DKIM authentication for an email identity | Write | |||
| PutEmailIdentityDkimSigningAttributes | Grants permission to configure or change the DKIM authentication settings for an email domain identity | Write | |||
| PutEmailIdentityFeedbackAttributes | Grants permission to enable or disable feedback forwarding for an email identity | Write | |||
| PutEmailIdentityMailFromAttributes | Grants permission to enable or disable the custom MAIL FROM domain configuration for an email identity | Write | |||
| PutSuppressedDestination | Grants permission to add an email address to the suppression list | Write | |||
| SendBulkEmail | Grants permission to compose an email message to multiple destinations | Write | |||
| SendCustomVerificationEmail | Grants permission to add an email address to the list of identities and attempts to verify it | Write | |||
| SendEmail | Grants permission to send an email message | Write | |||
| TagResource | Grants permission to add one or more tags (keys and values) to a specified resource | Tagging | |||
| TestRenderEmailTemplate | Grants permission to create a preview of the MIME content of an email when provided with a template and a set of replacement data | Write | |||
| UntagResource | Grants permission to remove one or more tags (keys and values) from a specified resource | Tagging | |||
| UpdateConfigurationSetEventDestination | Grants permission to update the configuration of an event destination for a configuration set | Write | |||
| UpdateContact | Grants permission to update a contact's preferences for a list | Write | |||
| UpdateContactList | Grants permission to update contact list metadata | Write | |||
| UpdateCustomVerificationEmailTemplate | Grants permission to update an existing custom verification email template | Write | |||
| UpdateEmailIdentityPolicy | Grants permission to update the specified sending authorization policy for the given identity (an email address or a domain) | Permissions management | |||
| UpdateEmailTemplate | Grants permission to update an email template | Write | |||
Resource types defined by Amazon Simple Email Service v2
The following resource types are defined by this service and can be used in the
Resource
element of IAM permission policy statements. Each action in the
Actions table
identifies the resource types that can be specified with that action. A resource type can also define which condition keys you can include in a policy. These keys are displayed in the last column of the Resource types table. For details about the columns in the following table, see
Resource types table
.
| Resource types | ARN | Condition keys |
|---|---|---|
| configuration-set |
arn:$
{
Partition}:ses:$
{
Region}:$
{
Account}:configuration-set/$
{
ConfigurationSetName}
|
|
| contact-list |
arn:$
{
Partition}:ses:$
{
Region}:$
{
Account}:contact-list/$
{
ContactListName}
|
|
| custom-verification-email-template |
arn:$
{
Partition}:ses:$
{
Region}:$
{
Account}:custom-verification-email-template/$
{
TemplateName}
|
|
| dedicated-ip-pool |
arn:$
{
Partition}:ses:$
{
Region}:$
{
Account}:dedicated-ip-pool/$
{
DedicatedIPPool}
|
|
| deliverability-test-report |
arn:$
{
Partition}:ses:$
{
Region}:$
{
Account}:deliverability-test-report/$
{
ReportId}
|
|
| identity |
arn:$
{
Partition}:ses:$
{
Region}:$
{
Account}:identity/$
{
IdentityName}
|
|
| import-job |
arn:$
{
Partition}:ses:$
{
Region}:$
{
Account}:import-job/$
{
ImportJobId}
|
|
| template |
arn:$
{
Partition}:ses:$
{
Region}:$
{
Account}:template/$
{
TemplateName}
|
Condition keys for Amazon Simple Email Service v2
Amazon Simple Email Service v2 defines the following condition keys that can be used in the
Condition
element of an IAM policy. You can use these keys to further refine the conditions under which the policy statement applies. For details about the columns in the following table, see
Condition keys table
.
To view the global condition keys that are available to all services, see Available global condition keys .
| Condition keys | Description | Type |
|---|---|---|
| aws:RequestTag/${TagKey} | Filters actions based on the presence of tag key-value pairs in the request | String |
| aws:ResourceTag/${TagKey} | Filters actions based on tag key-value pairs attached to the resource | String |
| aws:TagKeys | Filters actions based on the presence of tag keys in the request | ArrayOfString |
| ses:ApiVersion | Filters actions based on the SES API version | String |
| ses:FeedbackAddress | Filters actions based on the "Return-Path" address, which specifies where bounces and complaints are sent by email feedback forwarding | String |
| ses:FromAddress | Filters actions based on the "From" address of a message | String |
| ses:FromDisplayName | Filters actions based on the "From" address that is used as the display name of a message | String |
| ses:Recipients | Filters actions based on the recipient addresses of a message, which include the "To", "CC", and "BCC" addresses | ArrayOfString |


Many times I see abuse of the Return Value in a stored procedure. What I mean by this is essentially using the Return Value to send back say an identity value or something like:
SET @TheID = SCOPE_IDENTITY();
RETURN @TheID
My personal thought on the matter is that you use the Return Values are for determining the execution health of the procedure you called, and that to get an ID value either from a record set or an OUTPUT parameter. This allows a separation of concerns and provides a way for an application to completely understand the intent of the procedure that executed. Lets jump to the code to see what I mean:
--drop table dbo.pizza
CREATE TABLE Pizza (pizzaid TINYINT IDENTITY(1,1), pizzasize VARCHAR(10), numberoftoppings TINYINT, pizzaname VARCHAR(40) )
CREATE PROCEDURE dbo.asp_InsertNewPizza
(@PizzaSize VARCHAR(10),@NumberofToppings TINYINT,@pizzaName VARCHAR(40),@PizzaID TINYINT OUTPUT
AS
BEGIN TRY
INSERT INTO dbo.Pizza (pizzaSize, NumberOftoppings,pizzaname)
VALUES (@PizzaSize, @NumberofToppings, @pizzaName)
SELECT @PizzaID = SCOPE_IDENTITY();
RETURN 0;
END Try
BEGIN Catch
RETURN -1
END Catch
GO
DECLARE @InsertedID TINYINT, @RunOK INT;
EXEC @RunOK = dbo.asp_InsertNewPizza @pizzaSize = 'Medium',@NumberOfToppings=2, @PizzaName='Medium Cheese and Sausage', @PizzaID=@InsertedID OUT
SELECT @RunOK, @InsertedID
So, when we run the above code, we get a 0 for the return value (@RunOK), and a 1 for the inserted ID. Had we received a -1 for the return value, we would know that something was jacked up and not to trust the value of the OUTPUT param. Of course, like many demo’s, this is an incredibly simple demo just to show proof of concept. We could go a huge discourse about handling failures inside the procedure, application level transactions, or any number of scenarios that might result in a “Fine, in that case don’t do the above” but all that does is take away from the general message of separation of concerns and being able to quickly convey what actually happened. Lets look at this slight alteration to our operation:
CREATE PROCEDURE dbo.asp_InsertNewPizza
(@PizzaSize VARCHAR(10),@NumberofToppings TINYINT,@pizzaName VARCHAR(40),@PizzaID TINYINT OUTPUT
AS
IF EXISTS (SELECT pizzaID FROM dbo.Pizza WHERE pizzaname = @pizzaName)
BEGIN
SET @PizzaID = NULL;
RETURN 1;
END
ELSE
BEGIN TRY
INSERT INTO dbo.Pizza (pizzaSize, NumberOftoppings,pizzaname)
VALUES (@PizzaSize, @NumberofToppings, @pizzaName)
SELECT @PizzaID = SCOPE_IDENTITY();
RETURN 0;
END TRY
BEGIN CATCH
RETURN -1
END CATCH
GO
Without catching the Return Values… if we get a NULL for the identity, how do we know if it is from an issue with inserting the values or an issue with a pizza name already existing? If we capture the ReturnValue and see that it is >= 0 we know that it wasn’t from an error, but that a certain business rule had been met.
Well this is all good and well to us the database people, but what do we do if the app devs say it is too hard to won’t help them. Here is just a basic call in C# with ADO.NET and how it could be used:
SqlCommand databaseCMD = new SqlCommand ("asp_InsertNewPizza", pizzaConn);
databaseCMD.CommandType = CommandType.StoredProcedure;
SqlParameter PizzaSize = databaseCMD.Parameters.Add ("@pizzaSize", SqlDbType.VarChar, 10);
PizzaSize.Direction = ParameterDirection.Input;
SqlParameter NumberOfToppings = databaseCMD.Parameters.Add ("@NumberOfToppings", SqlDbType.TinyInt);
NumberOfToppings.Direction = ParameterDirection.Input;
SqlParameter PizzaName = databaseCMD.Parameters.Add ("@PizzaName", SqlDbType.VarChar, 40);
PizzaName.Direction = ParameterDirection.Input;
SqlParameter PizzaID = databaseCMD.Parameters.Add ("@PizzaID", SqlDbType.TinyInt);
PizzaID.Direction = ParameterDirection.Output ;
SqlParameter ReturnValue = databaseCMD.Parameters.Add ("RetVal", SqlDbType.Int);
RetVal.Direction = ParameterDirection.ReturnValue;
int pizzaID;
pizzaConn.Open();
pizzaID =databaseCMD.ExecuteScalar ().ToString() ;
if (ReturnValue == 0){
ResponseWrite("The Pizza was added. The pizzaID is : " + pizzaID.ToString() );
ResponseWreite("Return Value: " + RetVal.Value);
}
else if{ ReturnValue == 1) {
ResponseWrite("Sorry friend, that pizza name already exists.");
ResponseWreite("Return Value: " + RetVal.Value);
}
else { FreakOut(); }


Toggle navigation
Alphabets Heaven
Audio
Video
Releases
Everything Stays The Same by Alphabets Heaven
Lovers by Alphabets Heaven
Mountainsound by Alphabets Heaven
Siamese Burn EP by Alphabets Heaven
Rosewater by Alphabets Heaven
Boosh by Alphabets Heaven
Jay's Odyssey by Alphabets Heaven


EFA enquiries
Contact form https://form.education...
For all enquiries for the Education Funding Agency
Guidance
Protection for specialist provision in specialist post-16 institutions and in eligible further education (FE) institutions finished at the end of the 2015 to 2016 academic year. Please visit our post-16 funding pages for the latest funding information.
This document sets out how transitional protection will work and gives examples of how institutions may be protected.


Small but irritating things - Git.
18 Jul 2018I tried to retrieve a stash by name in Git. Honest, thats all I did.
Yet here I am.
 Repeat after me - Software is full of small and intensely irritating things.
You will encounter it again and again, in good software, bad software, throw away software and never used software (for obvious reasons). And you will encounter it in the best of software too.
Which, from now onwards, I will be documenting.
And today is a Git day.
Short historical background on GIT.
Lets digress -
- The person who wrote Git was, or rather, is the most famous programmer alive and quite possibly, the most respected as well. Name's Linus Torvalds.
- Git again is like a cross between the Mona Lisa and the Swiss Army Knife (the sort which Schwarzenneger carries into a Predator jungle and builds a seven storeyed Harrods with). Its an engineering marvel; Considering the distributed part, its a conceptual marvel; its incredibly robust; It's blazing fast and its very versatile.
So given these ancestors and these antecedents, you would assume that Git is flawless - right? And its venerable primary author and the plethora of talented programmers who have lovingly grafted their art onto the foundations laid by the erstwhile venerable one, have paid complete attention to both functionality and ease of use which is why you have a billion flags (options) with each command, and there are at least 7 ways to do any one thing (exaggerating only a bit), and its rock solid - correct? Well, yes, its all of that, but inspite of all that try, just try, to retrieve a stash by name.
Small but intensely irritating.
Here is the small but intensely irritating part - there is no easy way to do it.. The best you can do is find the first one that matches a RegEx.
That's odd (and I am being polite here).
In Git, arguably one of the best softwares on the planet, you can easily add a key-value pair, but you cannot easily retrieve it by key.
What?
Let me rephrase this - You are using an excellent programming language. Probably the best ever. In that language there is a dictionary or hashtable or map implementation. It allowes you to stuff things into a key-value collection by a key (random string). But here's the catch, it does not give you way to retrieve the same by name. Not easily anyway.
- Now list down your opinion of the programmer, designer and architect of the person who did this. And then...
- Map the attributes you wrote down against the names Linus Torvalds or Git.
(The second bullet point above would probably result in something resembling that OCD nightmare meme - a leaning tower of Pisa in a square box in a square wall picture frame. IT. DOES. NOT. COMPUTE. How did these people miss this?)
Software has issues. Period.
Ok, hyperbole apart, this is basically what software is.
Take the best software, built by the best people - It doesn't matter. Something, somewhere is inevitably going to fall off the radar, or the planning, or the execution, or the design.
In this case Git is probably the best DVCS out there, conceived by a marquee brand leader, built and maintained over many years by a great team, yet the fact that if you stored something by name, you would want to retrieve it by name seems to have escaped all those people who participated in it over all the years.
And, as is usually the case, it never became important enough to get put back in.
So, Git Unstash.
Yeah, ok software got issues. But we got developers. And developers fix issues. And what's a technical blog post without a solution to an issue anyway?
So here's how to do unstash in Git. (even though its available just a couple of google clicks away).
Git - stash by name.
This works just so -
git stash save "thisName"
The above is deprecated but still works. The current recommended way is -
git stash push "thisName"
.
Git - unstash by name.
This is the iffy thing. you can do this -
git stash apply stash^{/thisNa}
Notice the 'thisNa', that's a regex which contains part of your stash message. Additionally, what this does is retrieve the first named stash value which matches. Which means, if you do this
git stash save 'thisName1'
and this
git stash save 'thisName2'
,
You are out of luck during retrieval unless you can remember the exact second phrase, like this -
git stash apply stash^{/thisName2}"
,
Since Git returns only the first match, using part of the phrase leads you to the first partial match which may or may not be what you want.
There are alternatives which involve using a combination of
git stash list
and
git stash pop/apply stash@{n}
where n is a number referring to nth stash from top, etc (and use either pop or apply, not both).
For additional tips and sources and other more detailed knowhow, look up this SO question.
Bottomline.
Software, even the best software, has nuances and idiosyncrasies. Stop cribbing, drink your medicine, and get back to work(arounds).


| OpenID Connect Credential Provider | April 2021 | |
| Looker, et al. | Informational | [Page] |
OpenID Connect 1.0 is a simple identity layer on top of the OAuth 2.0 protocol. It enables relying parties to verify the identity of the End-User based on the authentication performed by an Authorization Server, as well as to obtain basic profile information about the End-User.¶
OpenID Providers today within OpenID Connect assume many roles, one of these is providing End-User claims to the relying party at the consent of the End-User such as their name or date of birth, providers performing this function are often referred to as being claims providers. However, the need for End-Users to be able to provide a variety of claims to a relying party from different providers is only increasing as many business processes that span multiple logical domains such as KYC and education move towards digital channels.¶
However, assuming a direct integration between the relying party and the claims providers leads to a difficult experience for End-Users to manage. Instead End-Users need a way to consolidate the different identities and claims they have available with various claims providers into one place where they can manage their release from. In doing this, a layer of in-direction is created between the relying party and the claims provider through the introduction of a new party that we refer to in this specification as being the "holder".¶
In OpenID Connect today the existing ways to communicate End-User claims to relying parties are the id_token and the userinfo endpoint, however these mechanisms alone are unsuitable for the style of indirect presentation of claims to relying parties via a holder, as the relying party must be able to authenticate the authority of the holder to be presenting the claims on behalf of the End-User. Instead in order to support this style of flow, this specification defines a new vehicle for communicating End-User claims called a "credential". In addition to this definition this specification defines how an existing OpenID Provider can be extended to issue "credentials" to holders.¶
OpenID Connect 1.0 is a simple identity layer on top of the OAuth 2.0 protocol. It enables relying parties to verify the identity of the End-User based on the authentication performed by an Authorization Server, as well as to obtain basic profile information about the End-User.¶
OpenID Providers today within OpenID Connect assume many roles, one of these is providing End-User claims to the relying party at the End-User's consent such as their name or date of birth. OpenID providers performing this function are often referred to as being claims providers. However, the need for End-Users to be able to provide a variety of claims from different providers is only increasing as many business processes that span multiple logical domains such as KYC and education move towards digital channels.¶
However, assuming a direct integration between the relying party and the claims providers leads to a difficult experience for End-Users to manage. Instead End-Users need a way to consolidate the different identities and claims they have available with various claims providers into one place where they can manage their release from. In doing this, a layer of indirection is created between the relying party and the claims provider through the introduction of a new party that we refer to in this specification as being the "holder".¶
In OpenID Connect today the existing ways to communicate End-User claims to relying parties are the id_token and the userinfo endpoint. However, these mechanisms alone are unsuitable for the style of indirect presentation of claims to relying parties via a holder as the relying party must be able to authenticate the authority of the holder to be presenting the claims on behalf of the End-User. Instead, in order to support this style of flow, this specification defines a new vehicle for communicating End-User claims called a "credential". In addition to this definition this specification defines how an existing OpenID Provider can be extended to issue "credentials" to holders.¶
To reiterate, this specification defines a protocol where a Credential Issuer (OP) may provide a Credential (set of claims) to a Credential Holder (acting as an RP) of which that Credential Holder (acting as an OP) controls and may present onward to Verifiers (RPs).¶
Note that the protocol for a Credential Holder to present a Credential to a Credential Verifier is outside the scope of this specification.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 @!RFC2119
.¶
In the .txt version of this document, values are quoted to indicate that they are to be taken literally. When using these values in protocol messages, the quotes MUST NOT be used as part of the value. In the HTML version of this document, values to be taken literally are indicated by the use of this fixed-width font.¶
All uses of JSON Web Signature (JWS) JWS and JSON Web Encryption (JWE) JWE data structures in this specification utilize the JWS Compact Serialization or the JWE Compact Serialization; the JWS JSON Serialization and the JWE JSON Serialization are not used.¶
This specification uses the terms defined in OpenID Connect Core 1.0; in addition, the following terms are also defined:¶
A set of claims about the End-User (subject) which is cryptographically bound to the credential holder in an authenticatable manner based on public/private key cryptography.¶
An OpenID Connect Authentication Request that results in the End-User being authenticated by the Authorization Server and the Client (Credential Holder) receiving a credential about the authenticated End-User.¶
A role an entity performs by holding credentials and presenting them to RPs on behalf of the consenting End-User (subject of the credentials). A Credential Holder serves the role of an OP (when presenting credentials to RPs) and an RP (when receiving credentials from Credential Issuers).¶
A role an entity performs by asserting claims about one or more subjects, creating a credential from these claims (cryptographically binding them to the holder), and transmitting the credential to the Credential Holder. A Credential Issuer is an OP that has been extended in order to also issue credentials.¶
An entity about which claims are made. Example subjects include human beings, animals, and things. In many cases the Credential Holder of a credential is the subject, but in certain cases it is not. For example, a parent (the Credential Holder) might hold the credentials of a child (the subject), or a pet owner (the Credential Holder) might hold the credentials of their pet (the subject). Most commonly the subject will be the End-User.¶
A role an entity performs by receiving one or more credentials for processing. A verifier is an RP that has been extended to receive and process credentials.¶
This specification extends the OpenID Connect protocol for the purposes of credential issuance. Whereby credential issuance refers to a protocol where a Credential Holder (acting as an RP) makes a request to a Credential Issuer (acting as an OP) to have a credential issued to it so that it may at a later stage then present this credential at the consent of the end user to Credential Verifiers (RPs). The steps in the credential issuance protocol are as follows:¶
The Credential Holder (acting as an RP) sends a Credential Request to the Credential Issuer (acting as an OP).¶
The Credential Issuer authenticates the End-User and obtains authorization.¶
The Credential Issuer responds with a Credential.¶
These steps are illustrated in the following diagram:¶
+----------+ +----------+ | | | | | |---(1)OpenID Credential Request--->| | | | | | | | +--------+ | | | | | | | | |Credential| | End- |<--(2) AuthN & AuthZ-->|Credential| | Holder | | User | | Issuer | | (RP) | | | | (OP) | | | +--------+ | | | | | | | |<--(3)OpenID Credential Response---| | | | | | +----------+ +----------+¶
Note - Outside of the scope for this specification is how the Credential Holder then exercises presentation of this credential with a Credential Verifier, however the diagram looks like the following.¶
The Credential Verifier (acting as a relying party) sends an OpenID Request to the Credential Holder (acting an OP).¶
The Credential Holder authenticates the End-User and obtains authorization.¶
The Credential Holder responds with a Credential Presentation.¶
+----------+ +----------+ | | | | | |---(1)OpenID Connect Credential Presentation Request------>| | | | | | | | +--------+ | | | | | | | | |Credential| | End- |<--(2) AuthN & AuthZ-->|Credential| | Verifier | | User | | Holder | | (RP) | | | | (OP) | | | +--------+ | | | | | | | |<--(3)OpenID Connect Credential Presentation Response------| | | | | | +----------+ +----------+¶
A Credential Request is an OpenID Connect authentication request made by a Credential Holder that requests the End-User to be authenticated by the Credential Issuer and consent be granted for a credential containing the requested claims about the End-User be issued to it.¶
The following section outlines how an OpenID Connect Authentication Request is extended in order to become a valid Credential Request.¶
The simplest OpenID Connect Credential Request is an ordinary OpenID Connect request that makes use of one additional scope, openid_credential
.¶
A non-normative example of the Credential Request.¶
HTTP/1.1 302 Found Location: https://server.example.com/authorize? response_type=code &scope=openid%20openid_credential &client_id=s6BhdRkqt3 &state=af0ifjsldkj &redirect_uri=https%3A%2F%2Fclient.example.org%2Fcb &credential_format=w3cvc-jsonld¶
When a request of this nature is made, the access_token
issued to the Credential Holder authorizes it to access the credential endpoint to obtain a credential from the Credential Issuer.¶
A Credential Request uses the OpenID and OAuth2.0 request parameters as outlined in section <IP_ADDRESS> of OpenID Connect core, except for the following additional constraints.¶
REQUIRED. A Credential Request MUST contain the openid_credential
scope value in the second position directly after the openid
scope.¶
REQUIRED. Determines the format of the credential returned at the end of the flow, values supported by the OpenID Provider are advertised in their openid-configuration metadata, under the credential_formats_supported
attribute.¶
OPTIONAL. Used when making a Signed Credential Request, defines the key material the Credential Holder is requesting the credential to be bound to and the key responsible for signing the request object. The value is a JSON Object that is a valid JWK.¶
OPTIONAL. Defines the relationship between the key material the Credential Holder is requesting the credential to be bound to and a decentralized identifier. Processing of this value requires the CI to support the resolution of decentralized identifiers which is advertised in their openid-configuration metadata, under the dids_supported
attribute. The value of this field MUST be a valid decentralized identifier.¶
Public private key pairs are used by a requesting Credential Holder to establish a means of binding to the resulting credential. A Credential Holder making a Credential Request to a Credential Issuer must prove control over this binding mechanism during the request, this is accomplished through the extended usage of a signed request defined in OpenID Connect Core.¶
It is RECOMMENDED that a Credential Request flow use the authorization code flow as defined in OpenID Connect core.¶
Successful and Error Authentication Response are in the same manor as OpenID Connect Core 1.0 with the code
parameter always being returned with the Authorization Code Flow.¶
On Request to the Token Endpoint the grant_type
value MUST be authorization_code
inline with the Authorization Code Flow and the code
value included as a parameter.¶
The following is a non-normative example of a response from the token endpoint, whereby the access_token
authorizes the Credential Holder to request a credential
from the credential endpoint.¶
{ "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6Ikp..sHQ", "token_type": "bearer", "expires_in": 86400, "id_token": "eyJodHRwOi8vbWF0dHIvdGVuYW50L..3Mz" }¶
The Credential Endpoint is an OAuth 2.0 Protected Resource that when called, returns Claims about the authenticated End-User in the form of a credential. To obtain a credential on behalf of the End-User, the Credential Holder makes a request to the Credential Endpoint using an Access Token obtained through OpenID Connect Authentication whereby the the openid_credential
scope was granted.¶
Communication with the Credential Endpoint MUST utilize TLS. See Section 16.17 for more information on using TLS.¶
The Credential Endpoint MUST support the use of HTTP POST methods defined in RFC 2616 [RFC2616].¶
It is recommended that the Credential Endpoint SHOULD enforce presentation of the OAuth2.0 Access Token to be sender constrained DPOP. However the Credential Endpoint MAY also accept Access Tokens as OAuth 2.0 Bearer Token Usage [RFC6750].¶
The Credential Endpoint SHOULD support the use of Cross Origin Resource Sharing (CORS) [CORS] and or other methods as appropriate to enable Java Script Clients to access the endpoint.¶
The Credential Holder may provide a signed request object containing the sub
to be used as the subject for the resulting credential. When a sub
claim is present within the request object an associated sub_jwk
claim MUST also be present of which the request object MUST be signed with, therefore proving control over the sub
.¶
The Credential Holder may also specify the credential_format
they wish the returned credential to be formatted as. If the Credential Issuer receiving the request does not support the requested credential format they it MUST return an error response, as per [TODO]. If the credential_format
is not specified in the request the Credential Issuer SHOULD respond with their preferred or default format. (Note if we are going to have a default we need to specify it or is it at the discretion of the Credential Issuer to determine this?)¶
When a signed request is not provided the Credential Issuer will use the sub
associated with the initial Credential request, where possible. If a sub
value is not available the Credential Issuer MUST return an error response, as per [TODO].¶
OPTIONAL. A valid OIDC signed JWT request object. The request object is used to provide a sub
the Credential Holder wishes to be used as the subject of the resulting credential as well as provide proof of control of that sub
.¶
A non-normative example of a Signed Credential request.¶
POST /credential HTTP/1.1 Host: https://issuer.example.com Authorization: Bearer <access-token> Content-Type: application/json { "request": <signed-jwt-request-obj> }¶
Where the decoded payload of the request parameter is as follows:¶
{ "aud": "https://issuer.example.com", "iss": "https://wallet.example.com", "sub": "urn:uuid:dc000c79-6aa3-45f2-9527-43747d5962a5", "sub_jwk" : { "crv":"secp256k1", "kid":"YkDpvGNsch2lFBf6p8u3", "kty":"EC", "x":"7KEKZa5xJPh7WVqHJyUpb2MgEe3nA8Rk7eUlXsmBl-M", "y":"3zIgl_ml4RhapyEm5J7lvU-4f5jiBvZr4KgxUjEhl9o" }, "credential_format": "w3cvc-jwt", "nonce": "43747d5962a5", "iat":<PHONE_NUMBER>, "exp":<PHONE_NUMBER> }¶
format : REQUIRED. The proof format the credential was returned in. For example w3cvc-jsonld
or w3cvc-jwt
.
credential : REQUIRED. A cryptographically verifiable proof in the defined proof format
. Most commonly a Linked Data Proof or a JWS.¶
{ "format": "w3cvc-jsonld", "credential": <credential> }¶
Formats of the credential
can vary, examples include JSON-LD or JWT based Credentials, the Credential Issuer SHOULD make their supported credential formats available at their openid-configuration metadata endpoint.¶
The following is a non-normative example of a Credential issued as a W3C Verifiable Credential 1.0 compliant format in JSON-LD.¶
{ "@context": [ "https://www.w3.org/2018/credentials/v1", "https://www.w3.org/2018/credentials/examples/v1" ], "id": "http://example.gov/credentials/3732", "type": ["VerifiableCredential", "UniversityDegreeCredential"], "issuer": "did:key:z6MkjRagNiMu91DduvCvgEsqLZDVzrJzFrwahc4tXLt9DoHd", "issuanceDate": "2020-03-10T04:24:12.164Z", "credentialSubject": { "id": "urn:uuid:dc000c79-6aa3-45f2-9527-43747d5962a5", "jwk": { "crv":"secp256k1", "kid":"YkDpvGNsch2lFBf6p8u3", "kty":"EC", "x":"7KEKZa5xJPh7WVqHJyUpb2MgEe3nA8Rk7eUlXsmBl-M", "y":"3zIgl_ml4RhapyEm5J7lvU-4f5jiBvZr4KgxUjEhl9o" }, "givenName": "John", "familyName": "Doe", "degree": { "type": "BachelorDegree", "name": "Bachelor of Science and Arts" } }, "proof": { "type": "Ed25519Signature2018", "created": "2020-04-10T21:35:35Z", "verificationMethod": "did:key:z6MkjRagNiMu91DduvCvgEsqLZDVzrJzFrwahc4tXLt9DoHd#z6MkjRagNiMu91DduvCvgEsqLZDVzrJzFrwahc4tXLt9DoHd", "proofPurpose": "assertionMethod", "jws": "eyJhbGciOiJFZERTQSIsImI2NCI6ZmFsc2UsImNyaXQiOlsiYjY0Il19..l9d0YHjcFAH2H4dB9xlWFZQLUpixVCWJk0eOt4CXQe1NXKWZwmhmn9OQp6YxX0a2LffegtYESTCJEoGVXLqWAA" } }¶
The following is a non-normative example of a Credential issued as a JWT¶
ewogICJhbGciOiAiRVMyNTYiLAogICJ0eXAiOiAiSldUIgp9.ewogICJpc3MiOiAiaXNzdWVyIjogImh0dHBzOi8vaXNzdWVyLmVkdSIsCiAgInN1YiI6ICJkaWQ6ZXhhbXBsZToxMjM0NTYiLAogICJpYXQiOiAxNTkxMDY5MDU2LAogICJleHAiOiAxNTkxMDY5NTU2LAogICJodHRwczovL3d3dy53My5vcmcvMjAxOC9jcmVkZW50aWFscy9leGFtcGxlcy92MS9kZWdyZWUiOiB7CiAgICAgImh0dHBzOi8vd3d3LnczLm9yZy8yMDE4L2NyZWRlbnRpYWxzL2V4YW1wbGVzL3YxL3R5cGUiOiAiQmFjaGVsb3JEZWdyZWUiLAogICAgICJodHRwczovL3d3dy53My5vcmcvMjAxOC9jcmVkZW50aWFscy9leGFtcGxlcy92MS9uYW1lIjogIkJhY2hlbG9yIG9mIFNjaWVuY2UgYW5kIEFydHMiCiAgfQp9.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c¶
And the decoded Claim Set of the JWT¶
{ "iss": "https://issuer.example.com", "sub": "urn:uuid:dc000c79-6aa3-45f2-9527-43747d5962a5", "sub_jwk" : { "crv":"secp256k1", "kid":"YkDpvGNsch2lFBf6p8u3", "kty":"EC", "x":"7KEKZa5xJPh7WVqHJyUpb2MgEe3nA8Rk7eUlXsmBl-M", "y":"3zIgl_ml4RhapyEm5J7lvU-4f5jiBvZr4KgxUjEhl9o" }, "iat":<PHONE_NUMBER>, "exp":<PHONE_NUMBER>, "https://www.w3.org/2018/credentials/examples/v1/degree": { "https://www.w3.org/2018/credentials/examples/v1/type": "BachelorDegree", "https://www.w3.org/2018/credentials/examples/v1/name": "Bachelor of Science and Arts" } }¶
TODO improve this section¶
Decentralized identifiers are a resolvable identifier to a set of statements about the did subject including a set of cryptographic material (e.g public keys). Using this cryptographic material, a decentralized identifier can be used as an authenticatable identifier in a credential, rather than using a public key directly.¶
A Credential Holder submitting a signed Credential Request can request, that the resulting credential be bound to the Credential Holder through the usage of decentralized identifiers by using the did
field.¶
A Credential Holder prior to submitting a credential request SHOULD validate that the CI supports the resolution of decentralized identifiers by retrieving their openid-configuration metadata to check if an attribute of dids_supported
has a value of true
.¶
The Credential Holder SHOULD also validate that the CI supports the did method to be used in the request by retrieving their openid-configuration metadata to check if an attribute of did_methods_supported
contains the required did method.¶
A CI processing a credential request featuring a decentralized identifier MUST follow the following additional steps to validate the request.¶
Validate the value in the did
field is a valid decentralized identifier.¶
Resolve the did
value to a did document.¶
Validate that the key in the sub_jwk
field of the request is referenced in the authentication section of the DID Document.¶
If any of the steps fail then the CI MUST respond to the request with the Error Response parameter, section <IP_ADDRESS>. with Error code: invalid_did
.¶
The following is a non-normative example of requesting the issuance of a credential that uses a decentralized identifier.¶
{ "response_type": "code", "client_id": "IAicV0pt9co5nn9D1tUKDCoPQq8BFlGH", "sub_jwk" : { "crv":"secp256k1", "kid":"YkDpvGNsch2lFBf6p8u3", "kty":"EC", "x":"7KEKZa5xJPh7WVqHJyUpb2MgEe3nA8Rk7eUlXsmBl-M", "y":"3zIgl_ml4RhapyEm5J7lvU-4f5jiBvZr4KgxUjEhl9o" }, "did": "did:example:1234", "redirect_uri": "https://Client.example.com/callback", "credential_format": "w3cvc-jsonld" }¶
The following is a non-normative example of a credential endpoint response for the request shown above.¶
{ "format": "w3cvc-jsonld", "credential": { "@context": [ "https://www.w3.org/2018/credentials/v1", "https://www.w3.org/2018/credentials/examples/v1" ], "id": "http://example.gov/credentials/3732", "type": ["VerifiableCredential", "UniversityDegreeCredential"], "issuer": "https://issuer.edu", "issuanceDate": "2020-03-10T04:24:12.164Z", "credentialSubject": { "id": "did:example:1234", "degree": { "type": "BachelorDegree", "name": "Bachelor of Science and Arts" } }, "proof": { "type": "Ed25519Signature2018", "created": "2020-04-10T21:35:35Z", "verificationMethod": "https://issuer.edu/keys/1", "proofPurpose": "assertionMethod", "jws": "eyJhbGciOiJFZERTQSIsImI2NCI6ZmFsc2UsImNyaXQiOlsiYjY0Il19..l9d0YHjcFAH2H4dB9xlWFZQLUpixVCWJk0eOt4CXQe1NXKWZwmhmn9OQp6YxX0a2LffegtYESTCJEoGVXLqWAA" } } }¶
An OpenID provider can use the following meta-data elements to advertise its support for credential issuance in its openid-configuration defined by OpenID-Discovery.¶
credential_supported
Boolean value indicating that the OpenID provider supports the credential issuance flow.¶
credential_endpoint
A JSON string value indicating the location of the OpenID providers credential endpoint.¶
credential_formats_supported
A JSON array of strings identifying the resulting format of the credential issued at the end of the flow.¶
credential_claims_supported
A JSON array of strings identifying the claim names supported within an issued credential.¶
credential_name
A human readable string to identify the name of the credential offered by the provider.¶
dids_supported
Boolean value indicating that the OpenID provider supports the resolution of decentralized identifiers.¶
did_methods_supported
A JSON array of strings representing Decentralized Identifier Methods that the OpenID provider supports resolution of.¶
The following is a non-normative example of the relevant entries in the openid-configuration meta data for an OpenID Provider supporting the credential issuance flow¶
{ "dids_supported": true, "did_methods_supported": [ "did:ion:", "did:elem:", "did:sov:" ], "credential_supported": true, "credential_endpoint": "https://server.example.com/credential", "credential_formats_supported": [ "w3cvc-jsonld", "jwt" ], "credential_claims_supported": [ "given_name", "last_name", "https://www.w3.org/2018/credentials/examples/v1/degree" ], "credential_name": "University Credential" }¶
In certain instances it is advantageous to be able to construct a URL which points at an OpenID Connect provider, of which is invocable by a supporting OpenID Connect client.¶
The URL SHOULD use the scheme openid
to allow supporting clients to register intent to handle the URL.¶
The URL SHOULD feature the term discovery
in the host portion of the URL identifying the intent of the URL is to communicate discovery related information.¶
The URL SHOULD feature a query parameter with key issuer
who's value corresponds to a valid issuer identifier as defined in OpenID Connect Discovery. This identifier MUST be a url of the scheme https://
of which when concatenated with the string /.well-known/openid-configuration
and dereferenced by an HTTP GET request
results in the retrieval of the providers OpenID Connect Metadata.¶
The following is a non-normative example of an invocable URL pointing to the OpenID Provider who has the issuer identifier of https://issuer.example.com
¶
openid://discovery?issuer=https://issuer.example.com¶


--- layout: doc title: Documentation ---
The node you are looking for is at Script input to the shell.


I'M NOT RAPPAPORT A film review by Scott Renshaw Copyright 1997 Scott Renshaw
(Gramercy) Starring: Walter Matthau, Ossie Davis, Amy Irving, Craig T. Nelson, Boyd Gaines, Martha Plimpton. Screenplay: Herb Gardner. Producers: John Penotti, John Starke. Director: Herb Gardner. MPAA Rating: PG (profanity, violence) Running Time: 133 minutes. Reviewed by Scott Renshaw.
Writers are smacked around so often and with such impunity in Hollywood that it is perfectly understandable when a writer insists on directing his or her own material. It similarly makes sense, I suppose, for a playwright to want to direct the film adaptation of his own material, and Herb Gardner has been unusually protective of his plays. In 1986, he directed his own adaptation of THE GOODBYE PEOPLE; in 1996, he directed his own adaptation of I'M NOT RAPPAPORT. Clearly Gardner is attached to his creations. In fact, he is far too attached. Another director might have identified I'M NOT RAPPAPORT as a meandering play with one interesting idea, and given it the good tightening up it needed. Gardner clings to every line and situation, turning out a sluggish and repetitive meditation on aging.
Walter Matthau stars as Nat, a senior citizen in New York City who spends his days creating identities for himself and causing trouble for those he believes guilty of exploiting people. He is also troublesome to Midge Carter (Ossie Davis), the old gentleman he pesters with his tall tales every day on a Central Park bench. The two become entangled when Midge's failing eyesight threatens his job as an apartment building superintendent, and Nat offers unsolicited assistance by posing as Midge's attorney. Nat's daughter Clara (Amy Irving) considers her father's behavior grounds for having herself declared his guardian, but Nat refuses to change his ways, even when an attempt to con a drug dealer (Craig T. Nelson) places both Nat and Midge in danger.
I'M NOT RAPPAPORT is about a man's attempt to retain his dignity in spite of advancing years, and there are strong possibilities in the character of Nat. A committed Socialist and social agitator, Nat sees his deceptions in defense of the elderly as a natural extension of his life-long belief in the rights of the oppressed. The film's best scene is Nat's showdown with Danforth (Boyd Gaines), the representative from Midge's apartment building, as Nat browbeats the insincere yuppie into a realization that he is attempting to discard a human being who has out-lived his usefulness. Walter Matthau and Ossie Davis both give solid performances, obviously appreciating their own opportunities in rare leading roles, and there is an affecting sincerity to their comments about giving respect where it is due.
That scene between Nat, Midge and Danforth is noteworthy not just because it is the best one in I'M NOT RAPPAPORT, but also because it says everything Gardner has to say that is worth saying. The film basically consists of a series of conversations between characters, and virtually every one finds Nat trying to put something over on someone to prove he is still relevant or to save his skin. In one such scene, Nat challenges a young tough (Guillermo Diaz) who has been extorting protection money from seniors; in another, Nat tries to persuade his daughter that he does not need her supervision. The relationships are not compelling enough by themselves to carry those scenes, and the point that seniors are under-estimated and mis-treated is made and made and made again. As I'M NOT RAPPAPORT wanders through the last of its 133 minutes, you may begin to feel that you have been sitting in your seat as long as its main characters have been alive.
Gardner tries to make up for that repetitive structure by spicing up the third act with the drug dealer sub-plot, and what a strange encounter it is. In an attempt to protect a young woman (Martha Plimpton) who owes the dealer money, Nat plays the part of a stereotypical mob boss who warns the "Cowboy" dealer to leave her alone. What begins as broad comic relief soon turns nasty as the Cowboy (played with scary effectiveness by Nelson) proves smarter and more dangerous than Nat expected. Yet even this scene is a re-tread of the run-in with the young extortionist, as Nat pushes his act too far with someone unwilling to play along. Worse still, it leaves us with the depressing notion that the elderly should be seen and not heard. Gardner didn't need to take us through over two hours of hoops just so that we could end up right where we started. Well-intentioned and well-acted though it might be, I'M NOT RAPPAPORT is the product of a writer too married to every word, and a director too sparing with the knife.
On the Renshaw scale of 0 to 10 elder statements: 4.
Visit Scott Renshaw's MoviePage http://www-leland.stanford.edu/~srenshaw Subscribe to receive reviews directly via email See details on the MoviePage
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


We will create and deploy a sample application inside a Docker container. The containerized application will run as a single-instance and communicate with a Postgres database on the container-host’s network. In this example the Postgres database will not be running in a container.
As in the Jiffy with Docker and SQLite tutorial, our Dockerfile will be quite simple and we will go with some of the docker build defaults. These include allowing docker build to determine the ipv4 address to assign to the image, and also using the default Docker networking setup (–network-bridge).
Note that this is a setup for testing and development only. When deploying a jiffy-application with Docker, you should use a container clustering/management solution (Docker Swarm, Kubernetes, Rancher …). Doing will simplify application scaling and is also necessary in order to support the peer-to-peer networking requirement.
Ensure that Jiffy has been installed on your workstation by following the instructions provided in the Jiffy Installation section.
We will generate a new application called pgelibraryapp from a model file that contains a ‘Library’ and ‘Book’ entity. Two relationships are maintained in the model; a Library hasMany Books and a Book belongsTo a specific Library. An application generated from this model will allow many Jiffy application features to be tested.
Open a terminal window on your workstation and run jiffy using the Library-Book model file to generate the source-code for our test application as follows:
jiffy -m $GOPATH/src/github.com/1414C/jiffy/support/testing_models/hasManyBelongsTo.json -p /exp/pgelibraryapp
Remember that jiffys -p flag expects to be provided with an existing path underneath your $GOPATH/src folder. In the example invocation shown above, jiffy will create the pgelibraryapp folder underneath $GOPATH/src/exp/ and then write the generated source-code to this location.
Execution of the generator (jiffy) should result in output similar to:
2018/06/22 12:26:57 generated: /Users/stevem/gowork/src/exp/pgelibraryapp/models/librarym.go
2018/06/22 12:26:57 generated: /Users/stevem/gowork/src/exp/pgelibraryapp/models/librarym_ext.go
2018/06/22 12:26:57 generated: /Users/stevem/gowork/src/exp/pgelibraryapp/controllers/libraryc.go
...
...
...
2018/06/22 12:26:59 executing /usr/local/go/bin/goimports -w /Users/stevem/gowork/src/exp/pgelibraryapp/util/strings.go
2018/06/22 12:26:59 executing /usr/local/go/bin/gofmt -w /Users/stevem/gowork/src/exp/pgelibraryapp/util/strings.go
2018/06/22 12:26:59 executing /usr/local/go/bin/goimports -w /Users/stevem/gowork/src/exp/pgelibraryapp/appobj/appconf.go
2018/06/22 12:26:59 executing /usr/local/go/bin/gofmt -w /Users/stevem/gowork/src/exp/pgelibraryapp/appobj/appconf.go
Your output may look slightly different, particularly the database connection test which will almost certainly fail. This is nothing to be concerned about, as the generator is attempting to connect to a local Postgres instance using bad credentials.
The next step is to edit the generated application’s configuration files. Docker allocates a static ip-address for each container by default, and we will use that address in our application’s configuration file. ‘external_address’ refers to the address at which the application’s end-points will be available, while the ‘internal_address’ is used for cache updates and interprocess communication over web-socket connections. Strictly speaking, in a single-instance Docker deployment we could get away with maintaining the address-keys as follows:
.dev.config.json
{
"external_address": ":8080",
"internal_address": ":4444",
"env": "dev",
...
...
}
However, we will show how to obtain and insert the container’s real ip-address into the configuration file. This will be useful later if you deploy multiple application instances in Docker containers.
Open the generated .dev.config.json file in an editor and update the ‘external_address’ and ‘internal_address’ values with “xxx.xxx.xxx.xxx:8080” and “xxx.xxx.xxx.xxx:4444” respectively. Using an illegal ipv4 address as a placeholder/mask ensures that the container will not start in the event that the container’s address could not be assigned to the config keys. When you have finished, save the file after verifying that it looks like this:
.dev.config.json
{
"external_address": "xxx.xxx.xxx.xxx:8080",
"internal_address": "xxx.xxx.xxx.xxx:4444",
"env": "dev",
...
...
}
Jiffy generates configuration files targeting a Postgres database by default. Maintain the values in the ‘database’ block of the *.dev.config.json; file as necessary in order to gain access to your testing database:
.dev.config.json
{
...
"database": {
"db_dialect": "postgres",
"host": "<IP_ADDRESS>",
"port": 5432,
"usr": "godev",
"password": "gogogo123",
"name": "glrestgen"
},
...
}
Once the database block in .dev.config.json has been updated, save the file and close the editor.
By default, Postgres only allows connections from the localhost (<IP_ADDRESS>). If your Postgres database is running on another machine, you will need to ensure that it allows connections from remote hosts. A quick web-search will reveal how to maintain Postgres’s postgresql.conf and pg_hba.conf files to allow remote connections to the database. Make sure that you are able to connect to your Postgres database from the Docker host environment before continuing.
Jiffy generates configuration files supporting a stand-alone / local group-leadership KVS by default. Examine the ‘group_leader_kvs’ block in *.config.json to ensure the ‘local_standalone’ KVS option is active as shown below:
.dev.config.json
{
...
"group_leader_kvs": {
"local_standalone": {
"active": true,
"internal_address": "<IP_ADDRESS>:4444"
},
"redis": {
"active": false,
"max_idle": 80,
"max_active": 12000,
"redis_protocol": "tcp",
"redis_address": "<IP_ADDRESS>:6379"
},
"memcached": {
"active": false,
"memcached_addresses": [
"<IP_ADDRESS>:11211"
]
},
"sluggo": {
"active": false,
"sluggo_address": "<IP_ADDRESS>:7070"
}
},
...
}
Next, we need to build a static application binary for linux/amd64. To build the application, a number of dependencies are required as outlined in the Jiffy Dependencies section of the documentation. Check that your build environment contains the correct packages and import any that are missing.
This tutorial uses the popular Alpine Linux distribution as the basis for the new container image. Alpine Linux is a small and sparse distribution making it nice for use with containers. There are a few things to be aware of however…
Alpine Linux is based on lib-musl which means that binaries built by go build where the source makes use of cgo must target lib-musl rather than lib-gcc. The binary resulting from the typical ‘GOOS=linux GOARCH=amd64 go build -o main .’ command would almost certainly not work on Alpine. The good news is that it is quite easy to build with musl-gcc, the bad news is that musl-gcc is available for Linux only. If you are working on a system in which lib-musl is not supported, you will need to run the go build command in a supported build environment. Most Linux distributions and architectures are supported.
Check if lib-musl has been installed in the build environment by running the which command:
which musl-gcc
If musl-gcc was found in the $PATH, which will return output similar to:
/usr/bin/musl-gcc
If musl-gcc was not found, follow the installation instructions below to download and install the required packages.
We will go over how to install musl-gcc on a Debian system, but the steps are largely the same for any Linux distribution. Use your distribution’s package manager to install the musl lib, musl development files and musl development tools. Run the following commands (or their equivalents) in a terminal window on your Linux build system:
sudo apt-get update
sudo apt-get install musl
sudo apt-get install musl-dev
sudo apt-get install musl-tools
Check to make sure that musl-gcc is now in the $PATH:
which musl-gcc
After ensuring that all of the required dependencies have been installed in the build environment, run the following command to build a statically-linked binary called main for the Alpine Linux target OS. Setting the CC environment variable to point at musl-gcc ensures that the target (executable) will run in the Alpine Linux environment. Adjust the GOARCH environment variable as necessary:
CGO=0 GOOS=linux GOARCH=amd64 CC=$(which musl-gcc) go build --ldflags '-w -linkmode external -extldflags "-static"' -a -tags netgo -installsuffix cgo -o main .
Running go build with CGO=0 and setting the -a flag forces a rebuild without cross-compilation dependencies. Setting –ldflags as shown instructs go build to produce a statically linked binary. Setting the CC environment variable to point at musl-gcc ensures that the target (executable) will run in the Alpine Linux environment. Once the build has completed, a new ‘main’ file will have been created. Check the file via the file command:
file main
You should see output similar to:
main: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped
This is optional and will produce a marginally smaller binary. We will not use a dynamically-linked binary in our image, but have included a suitable go build command for reference purposes.
CGO=0 GOOS=linux GOARCH=amd64 CC=$(which musl-gcc) go build -a -tags netgo -installsuffix cgo -o main .
Running go build with CGO=0 and setting the -a flag forces a rebuild without cross-compilation dependencies. go build produces dynamically-linked binaries by default, so no linker instructions have been provided. Setting the CC environment variable to point at musl-gcc ensures that the target (executable) will run in the Alpine Linux environment. Once the build has completed, a new ‘main’ file will have been created. Check the file via the file command:
file main
You should see output similar to:
main: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, not stripped
Verify that you have Docker installed in your build environment. The easiest way to do this is to run which docker in a terminal window to ensure the docker application is in the $PATH. If there is no response, check the $PATH or install Docker following the instructions at www.docker.com. Verify that the Docker daemon is running by opening a terminal window in your build environment and running the docker ps command. If Docker is not running, an error message will be displayed. Start the Docker daemon before continuing.
In order to deploy the compiled application in a Docker container, we need to create a Dockerfile. The docker build command uses the Dockerfile as a set of instructions when building an image. As mentioned previously, we will use Alpine Linux as the foundation (base image) for the new Docker container image. Dockerhub has a number of pre-defined images that are available to be ‘pulled’ into locally defined custom images.
Create a new file called Dockerfile in the root folder of the libappmulti source-tree and open it in your editor. Copy the following content into the new Dockerfile. An effort has been made to briefly describe what each line of the Dockerfile is used for.
# use the official docker hub alpine:latest base image
FROM alpine:latest
# set the maintainer information for the new image
LABEL<EMAIL_ADDRESS># add the compiled application binary to the root folder of the new image
ADD main ./
# copy the configuration file to the root folder of the new image
COPY .dev.config.json .
# add the entrypoint.sh shell script to the root folder of the new image
ADD docker-entrypoint.sh .
# set widely exectuable permission on the shell-script
RUN /bin/chmod 777 docker-entrypoint.sh
# create a directory in the root folder of the new image to hold the jwt signing keys
RUN mkdir jwtkeys
# copy the jwtkeys folder content into the image's /jwtkeys folder
COPY jwtkeys ./jwtkeys
# set container environment variable $PORT to 8080
ENV PORT 8080
# container will listen on port tcp/8080
EXPOSE 8080
# install sqlite3 into the image
RUN apk update \
&& apk add sqlite \
&& apk add socat
# add unix file command
RUN apk add file
# create a test.db file in the root folder (don't really need to do this,
# but it is a nice test when getting started)
RUN /usr/bin/sqlite3 /test.db
# set the container entrypoint - container executes this once it is up and running
ENTRYPOINT ["/docker-entrypoint.sh"]
# specify the flag(s) to be used with the ENTRYPOINT
CMD ["-dev"]
In a previous step .dev.config.json was updated with xxx.xxx.xxx.xxx ipv4 address masks. In order to replace the masks with the container’s ipv4 address, docker run and docker start will execute the docker-entrypoint.sh when running a container instance based on the image definition. At the moment this is a problem however, as we have not written the script yet. Create a new file called docker-entrypoint.sh in the root folder of the pgelibraryapp source-tree and open it in your editor. Copy the following content into the new docker-entrypoint.sh file:
#!/bin/sh
# get the ipv4 address assigned to eth0
replace=$(ifconfig eth0 | grep "inet addr" | cut -d ':' -f 2 | cut -d ' ' -f 1)
# set a variable with the value we are planning to replace
search="xxx.xxx.xxx.xxx"
# check that variable replace has something in it
if [ -z "$replace" ]; then
echo "Did not get an address value for eth0"
elif [ -n "$replace" ]; then
echo "${replace} found"
# replace all instances of 'xxx.xxx.xxx.xxx' in .dev.config.json
# with the ipv4 address in the ${replace} variable
sed -i "s/${search}/${replace}/g" .dev.config.json
exec /main "$@"
fi
Note that the docker-entrypoint.sh script assumes the ipv4 address should be read from the eth0 interface. This may not be the case in more complex deployments.
Assuming the previous steps have been successful, it is now time to build the new Docker image. Execute the following command from the pgelibraryapp root folder:
docker build -t pgelibraryapp .
Running docker build as shown instructs Docker to construct an image called pgelibraryapp using the Dockerfile in the current working directory. You should see output similar to:
Sending build context to Docker daemon 14.4MB
Step 1/17 : FROM alpine:3.7
---> 3fd9065eaf02
Step 2/17 : LABEL<EMAIL_ADDRESS>---> Running in bf4ee48b2816
Removing intermediate container bf4ee48b2816
---> f6450f88abb9
Step 3/17 : ADD main ./
---> 0339472c223a
Step 4/17 : COPY .dev.config.json .
---> ea4d0e22dda8
Step 5/17 : ADD docker-entrypoint.sh .
---> 3a091afcc4e6
Step 6/17 : RUN /bin/chmod 777 docker-entrypoint.sh
---> Running in 02bc2611f675
Removing intermediate container 02bc2611f675
---> 82b131becf0a
Step 7/17 : RUN mkdir jwtkeys
---> Running in 5dc37bbb6d90
Removing intermediate container 5dc37bbb6d90
---> ce27be862527
Step 8/17 : COPY jwtkeys ./jwtkeys
---> fad638ee093f
Step 9/17 : ENV PORT 8080
---> Running in ebf5a4645a24
Removing intermediate container ebf5a4645a24
---> f73fa9102cca
Step 10/17 : EXPOSE 8080
---> Running in 8ec67da3a90f
Removing intermediate container 8ec67da3a90f
---> d943968d2257
Step 11/17 : RUN apk update
---> Running in 4f036d66ca8e
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
v3.7.0-215-g16971064c0 [http://dl-cdn.alpinelinux.org/alpine/v3.7/main]
v3.7.0-207-gac61833f9b [http://dl-cdn.alpinelinux.org/alpine/v3.7/community]
OK: 9054 distinct packages available
Removing intermediate container 4f036d66ca8e
---> 15aef1a3a835
Step 12/17 : RUN apk add file
---> Running in 7444e203db73
(1/2) Installing libmagic (5.32-r0)
(2/2) Installing file (5.32-r0)
Executing busybox-1.27.2-r7.trigger
OK: 9 MiB in 13 packages
Removing intermediate container 7444e203db73
---> 92bf177f86b7
Step 13/17 : RUN apk add busybox-extras
---> Running in 662591e9e2bd
(1/1) Installing busybox-extras (1.27.2-r11)
Executing busybox-extras-1.27.2-r11.post-install
Executing busybox-1.27.2-r7.trigger
OK: 9 MiB in 14 packages
Removing intermediate container 662591e9e2bd
---> 80ab4f590c82
Step 14/17 : RUN apk add openssh-client
---> Running in 5baaa52529ab
(1/2) Installing openssh-keygen (7.5_p1-r8)
(2/2) Installing openssh-client (7.5_p1-r8)
Executing busybox-1.27.2-r7.trigger
OK: 12 MiB in 16 packages
Removing intermediate container 5baaa52529ab
---> 8d1b14444e72
Step 15/17 : RUN apk add postgresql-client
---> Running in 2a3edc09549a
(1/9) Installing ncurses-terminfo-base (6.0_p20171125-r0)
(2/9) Installing ncurses-terminfo (6.0_p20171125-r0)
(3/9) Installing ncurses-libs (6.0_p20171125-r0)
(4/9) Installing libedit (20170329.3.1-r3)
(5/9) Installing db (5.3.28-r0)
(6/9) Installing libsasl (2.1.26-r11)
(7/9) Installing libldap (2.4.45-r3)
(8/9) Installing libpq (10.4-r0)
(9/9) Installing postgresql-client (10.4-r0)
Executing busybox-1.27.2-r7.trigger
OK: 24 MiB in 25 packages
Removing intermediate container 2a3edc09549a
---> 9d5c92cc037f
Step 16/17 : ENTRYPOINT ["/docker-entrypoint.sh"]
---> Running in 499c637488e1
Removing intermediate container 499c637488e1
---> 6b4c3914a599
Step 17/17 : CMD ["-dev"]
---> Running in 401321fde538
Removing intermediate container 401321fde538
---> 32045cc2b080
Successfully built 32045cc2b080
Successfully tagged pgelibraryapp:latest
Aardvark:pgelibraryapp stevem$
You can run the docker image command to view some basic data regarding the new image:
docker image ls pgelib*
You should see output similar to:
REPOSITORY TAG IMAGE ID CREATED SIZE
pgelibraryapp latest 32045cc2b080 About a minute ago 35MB
Once the image has been created, the next step is to use it to create a container. This can be done in a single step where Docker creates the container and then starts it. Run the following command to create and start a new container from the pgelibraryapp image:
docker run --name pgelibraryapp -p 8080:8080 -d pgelibraryapp
If all went well, docker created a container from the pgelibraryapp image and then started it. The flags provided with the docker run command do the following:
Container status can be checked via the docker ps command:
docker ps -a
You should see output similar to:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e04bbd64dfba pgelibraryapp "/main -dev" 3 seconds ago Up 3 seconds <IP_ADDRESS>:8080->8080/tcp pgelibraryapp
The output of the command shows that the pgelibraryapp container was started with entrypoint ‘/main -dev’ and that container port :8080 is mapped to host port :8080.
We can ssh into the running container to see what is going on using the command shown in the following terminal session. The command logs in as root using the specified interactive shell (in this case /bin/sh). Check that ‘main -dev’ is running as expected using the Linux ps command.
docker exec -it pgelibraryapp /bin/sh
/ # ps -ef
PID USER TIME COMMAND
1 root 0:00 /main -dev
12 root 0:00 /bin/sh
19 root 0:00 ps -ef
/ #
Take a look around the in the container environment and verify that the content of .dev.config.json has been updated with the correct ip-address using the ifconfig/cat commands. Remember that the docker container is created with a static ipv4 address assigned to the eth0 interface. Verify that the eth0 ipv4 address has been added to the ‘external_address’ and ‘internal_address’ keys in .dev.config.json.
/ # ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:AC:11:00:02
inet addr:<IP_ADDRESS> Bcast:<IP_ADDRESS> Mask:<IP_ADDRESS>
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:401 errors:0 dropped:0 overruns:0 frame:0
TX packets:275 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:61539 (60.0 KiB) TX bytes:26368 (25.7 KiB)
lo Link encap:Local Loopback
inet addr:<IP_ADDRESS> Mask:<IP_ADDRESS>
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:401 errors:0 dropped:0 overruns:0 frame:0
TX packets:401 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:70197 (68.5 KiB) TX bytes:70197 (68.5 KiB)
/ # cat .dev.config.json
{
"external_address": "<IP_ADDRESS>:8080",
"internal_address": "<IP_ADDRESS>:4444",
"env": "dev",
...
...
...
"service_name": "Book",
"service_active": true
}
]
}/ #
We installed the Postgres client package into the pgelibraryapp image by including it in our Dockerfile. Make sure that the pgelibraryapp container is running and that you have logged in via the docker exec -it pgelibraryapp /bin/sh command. Try to connect to your Postgres database as follows:
/ # psql -h <IP_ADDRESS> -d postgres -U postgres
Password for user postgres: *******************
psql (10.4, server 9.5.3)
Type "help" for help.
postgres=#
Connecting to the database using psql is just for demonstration purposes. If the database was not reachable the container would not have started. For a discussion of what to do if the container entrypoint fails, see the Troubleshooting section of this document.
Launch Postman or your favorite RESTful service testing tool and specify a target URL of: http://<IP_ADDRESS>:8080/usr/login
making sure to select the http POST method. Maintain the request body to provide a user-id and password as shown in the following JSON snippet. Typically the user-id for a jiffy application is an email address, but an exception is made for the default administration user definition.
{
"email": "admin",
"password": "initpass"
}
When you have finished and your Postman (or other test utility) looks like the following image, click the ‘Send’ button to post your login request to the running application.
We will create a new ‘Library’ entity. Configure your test-tool to POST to the ‘library’ service as shown in the following image:
Copy the JWT from the login session and paste it into the new POST request’s Authorization header field as shown below and then submit the request to the application.
Following submission, a new ‘Library’ entity should have been created:
Create another ‘Library’ entity using the ‘Create a Library’ steps and then request a list of Library entities using the GET ../librarys end-point:
Next, we will create a ‘Book’ entity and allocate it to ‘Library’ 1. Configure your test-tool to POST to the ‘book’ service as shown in the following image:
Following the submission, a new ‘Book’ entity should have been created:
Create a few more ‘Book’ entities using the ‘Create a Book’ steps and allocate them to your ‘Library’ entities. When you have finished, request a list of ‘Book’ entities using the GET ../books end-point:
Based on the ‘hasMany’ relationship between the ‘Library’ and ‘Book’ entity’s, we can get the number of ‘Book’ entities belonging to ‘Library’ 1 as follows:
Remove the $count suffix from the URL to get the complete list of Book entities belonging to ‘Library’ 1:
A reciprocal ‘belongsTo’ relationship exists between the ‘Book’ and ‘Library’ entity definitions. Select a ‘Book’ entity and verify that the ‘belongsTo’ relationship works as expected:
Check the list of filters and commands along with the Library and Book models to see what can be appended to the the service end-points. Try some things out.


Windows command line project management tool.
Makes it easy to switch between projects when using the Windows command shell. Each project exists in its own directory and changing the current project sets up environment variables for that project. For example, two software development projects can use different versions of Java or ant.
This tool is not limited to supporting software development projects; it can simply help you remember a list of projects and which directories each project's files are in. It is also easy to list all projects, quickly change projects, add new projects, edit the setup for an existing project, remove old projects, and show the current project.
chpj
name - changes the current project; runs its setup scriptlspj
- lists projectsadpj
name path - adds a projectcdpj
- changes to the current project's base directoryedpj
name - edits the setup script for a projecthlpj
- help, shows commandsshpj
- shows the current project name and pathrmpj
name path - removes a projectinpj
- installs the project switching scriptunpj
- uninstalls the project switching scriptin progress...
soon...
(c) 2011 M. Wright. All Rights Reserved.
Open Source, hosted at github, https://github.com/commandlinetools/winpjm; Apache License, Version 2.0


--- title: Israel shot 009 date: 2004-01-10T00:00:00.000Z filepathOriginal: photos/IMG0124.jpg filepathThumbnail: photos/IMG0124.thumbnail.jpg filepathPreview: photos/IMG0124.preview.jpg ---
Academy 2 chics, interspersed with the local Arab chics, at Shorashim.


6 Weighted Co-expression Network Analysis
The aims of this analysis include:
- Pre-process gene expression data for zebrafish brain RNA-seq dataset, and separately, for a human early-onset Alzheimer’s disease brain microarray dataset.
- Construct separate gene co-expression networks for zebrafish and human data.
- Identify modules of co-expressed genes in each network.
- Determine the biological significance of modules using functional enrichment analysis and by correlating module expression with sample traits/characteristics.

- Export network for visualisation.
6.1 Data Import and Pre-Processing
6.1.1 Zebrafish Data
Import the DGEList
object previous made in the DE gene analysis, geneCounts
. This object contains three data.frames, sample
(sample metadata), counts
(gene counts in cpm), and genes
(gene annotation from Biomart). The data has already been normalised (TMM method), which can be seen in Figure 6.1.
According to WGCNA’s FAQ page, genes can be filtered by mean expression of variance since low-expressed or non-varying genes usually represent noise, which is undesirable when forming a co-expression network. Below, we will filter by both mean expression and variance to retain genes which tend to be more highly expressed and variable across samples.
# Density plot of TMM-normalized gene expression distribution for each library before filtering.
A <- geneCounts %>%
cpm(log = TRUE) %>%
melt %>%
filter(is.finite(value)) %>%
ggplot(aes(x = value, colour = Var2)) +
geom_density() +
guides(colour = FALSE) +
ggtitle("Before filtering") +
labs(x = "logCPM", y = "Density")
# Retain genes which are expressed at sufficiently high levels, at least 1.5 cpm in 6 (half of the total number of) samples.
keepTheseGenes <- rowSums(cpm(geneCounts) > 1.5) >= 6
# Density plot of the gene expression distribution after filtering out low expressed genes.
B <- geneCounts %>%
cpm(log = TRUE) %>%
extract(keepTheseGenes,) %>%
melt %>%
filter(is.finite(value)) %>%
ggplot(aes(x = value, colour = Var2)) +
geom_density() +
guides(colour = FALSE) +
ggtitle("After filtering low exp genes")
# Retain genes with sufficiently high variation in expression across all samples.
# Here we define sufficiently high variation as having a standard deviation above the 25th quantile (so not in bottom 25%).
variableGenes <- rowSds(cpm(geneCounts, log=TRUE)) > quantile(rowSds(cpm(geneCounts, log=TRUE)), 0.25)
# Density plot of gene expression distribution after filtering out low variance genes.
C <- geneCounts %>%
cpm(log = TRUE) %>%
extract(variableGenes,) %>%
melt %>%
filter(is.finite(value)) %>%
ggplot(aes(x = value, colour = Var2)) +
geom_density() +
guides(colour = FALSE) +
ggtitle("After filtering variable genes")
# Density plot of gene expression distribution after filtering out both low expressed and low variance genes.
D <- geneCounts %>%
cpm(log = TRUE) %>%
extract(keepTheseGenes&variableGenes, ) %>%
melt %>%
filter(is.finite(value)) %>%
ggplot(aes(x = value, colour = Var2)) +
geom_density() +
guides(colour = FALSE) +
ggtitle("After filtering both")
# Plot the effect of filtering low expressed genes, low variation genes, and both.
grid.newpage()
print(A, vp = vp3)
print(B, vp = vp4)
print(C, vp = vp1)
print(D, vp = vp2)
After filtering out lowly expressed genes and low variance genes, each RNA-seq library contains 13381 genes. The distributions of gene counts in each RNA-seq library after normalization and filtering are shown in the following boxplots.
The WGCNA FAQ recommends performing a log2 transformation before transposing to get the matrix into the correct format for analysis. Here, we use edgeR’s cpm()
function with a log2
transformation, as the cpm()
function takes into account the TMM-normalisation factors previously calculated for each library. The final matrix will be stored in an object called zebrafishWGCNAGenes
.
zebrafishWGCNAGenes <- geneCounts %>% cpm(log=TRUE) %>% t %>% as.data.frame
6.1.2 Human Data
The data is Affymetrix Human Gene 1.1 ST Array gene (and transcript) expression data from human post-mortem individuals with and without PSEN1-based familial Alzheimer’s disease. In this dataset, there are 14 control individuals without Alzheimer’s disease or any neurodegeneration or comorbidities, and 7 individuals with familial Alzheimer’s disease due to a mutation in PSEN1 (1 with E120G
mutation, 2 with V89L
mutation, and 4 with V89L
mutation).
Processed and raw data can be downloaded from ArrayExpress under the accession number E-GEOD-39420
. The raw data has been processed through background adjustment, normalisation, and summarisation by Robust Multi-array Analysis (RMA) using the Affy package (version 1.30.0) from Bioconductor on R language (version 2.13.2). This method is considered reasonable, so we will proceed with this analysis using the processed gene expression data.
First we import the relevant files into R.
humanSamples
from the fileE-GEOD-39420.sdrf.txt
contains metadata (characteristics and comments) about each sample, including their age at death, APOE genotype, Braak stage (a morphological-based measure of neurodegeneration in their brain), sex, and importantly their PSEN1 genotype.humanExprs
contains the gene expression data for each sample (one sample per column and one probe per row), along with information related to each probe (row), including gene and transcript assignments.
In microarray data, there are multiple probes mapping to each gene We will use the collapseRows()
function from the WGCNA package to retain only one gene per probe, using the MaxMean
method to choose the gene which has the highest mean expression across the samples.
The boxplots below in Figure 6.4 show the gene expression distribution of each human sample.
We can also visualise the effect of restricting (filtering) the human dataset to include only the genes which were previously determined in the zebrafish dataset to have sufficiently high expression (1.5 cpm in at least 6 out of 12 samples) and variance (with standard deviation greater than the bottom 25% of standard deviations in the dataset). A comparison of the gene expression distribution before and after filtering is shown below.
6.2 Network Construction
6.2.1 Choice of Soft-Thresholding Power
The first step in constructing a scale-free gene co-expression requires choosing a soft-thresholding power.
Here, we test a range of possible soft-thresholding powers to find the minimum one which results in approximately scale-free topology for the zebrafish and human gene expression networks. We define an appropriate minimum scale-free topology fitting index R^2 to be at least 0.80.
# Choose a set of soft-thresholding powers to test.
powers = c(1:10, seq(from = 12, to = 20, by = 2))
# Call the network topology analysis function.
sft_human = pickSoftThreshold(human_genes,
powerVector = powers,
verbose = 0,
networkType = "signed",
RsquaredCut = 0.80)
## Power SFT.R.sq slope truncated.R.sq mean.k. median.k. max.k.
## 1 1 0.01010 3.000 0.839 4170.0 4190.0 4470
## 2 2 0.01110 0.972 0.965 2420.0 2410.0 2920
## 3 3 0.00481 -0.295 0.833 1530.0 1510.0 2140
## 4 4 0.01440 -0.243 0.667 1040.0 1000.0 1660
## 5 5 0.16300 -0.480 0.693 745.0 690.0 1360
## 6 6 0.53000 -0.812 0.858 554.0 492.0 1190
## 7 7 0.75500 -0.992 0.945 425.0 359.0 1060
## 8 8 0.87000 -1.100 0.974 335.0 267.0 958
## 9 9 0.91900 -1.160 0.983 270.0 202.0 875
## 10 10 0.94300 -1.190 0.984 221.0 155.0 804
## 11 12 0.97200 -1.240 0.990 156.0 94.9 693
## 12 14 0.97800 -1.270 0.985 115.0 60.1 609
## 13 16 0.98200 -1.280 0.985 87.7 39.5 543
## 14 18 0.98300 -1.280 0.983 68.9 26.9 489
## 15 20 0.98300 -1.270 0.981 55.4 18.7 443
sft_zebrafish = pickSoftThreshold(zebrafish_genes,
powerVector = powers,
verbose = 0,
networkType = "signed",
RsquaredCut = 0.80)
## Power SFT.R.sq slope truncated.R.sq mean.k. median.k. max.k.
## 1 1 0.170 2.79 0.553 4160.0 4180.0 4470
## 2 2 0.210 -3.58 0.550 2400.0 2340.0 2960
## 3 3 0.221 -2.31 0.726 1520.0 1470.0 2170
## 4 4 0.258 -1.79 0.832 1020.0 976.0 1680
## 5 5 0.354 -1.70 0.893 722.0 678.0 1340
## 6 6 0.421 -1.67 0.902 529.0 489.0 1100
## 7 7 0.495 -1.65 0.909 400.0 363.0 924
## 8 8 0.551 -1.65 0.907 310.0 274.0 785
## 9 9 0.597 -1.62 0.908 245.0 211.0 675
## 10 10 0.637 -1.61 0.906 197.0 165.0 587
## 11 12 0.742 -1.54 0.945 133.0 105.0 454
## 12 14 0.822 -1.55 0.977 93.4 70.1 365
## 13 16 0.841 -1.66 0.983 68.1 48.0 310
## 14 18 0.864 -1.72 0.990 51.1 33.9 267
## 15 20 0.881 -1.78 0.994 39.3 24.5 233
#-------------------------------------------------------------------------------------------#
# Plot the results:
#pdf("WGCNA/softThreshold.pdf", width = 11, height = 8)
par(mfrow = c(2,2));
cex1 = 0.9;
# Human: Scale-free topology fit index as a function of the soft-thresholding power
plot(sft_human$fitIndices[,1], -sign(sft_human$fitIndices[,3])*sft_human$fitIndices[,2],
xlab="Soft Threshold (power)",ylab="Scale Free Topology Model Fit,signed R^2",type="n",
main = paste("Scale independence"))
text(sft_human$fitIndices[,1], -sign(sft_human$fitIndices[,3])*sft_human$fitIndices[,2],
labels=powers,cex=cex1,col="red")
# this line corresponds to using an R^2 cut-off of h
abline(h=0.80,col="red")
# Mean connectivity as a function of the soft-thresholding power
plot(sft_human$fitIndices[,1], sft_human$fitIndices[,5],
xlab="Soft Threshold (power)",ylab="Mean Connectivity", type="n",
main = paste("Mean connectivity"))
text(sft_human$fitIndices[,1], sft_human$fitIndices[,5],
labels=powers, cex=cex1,col="red")
# Zebrafish: Scale-free topology fit index as a function of the soft-thresholding power
plot(sft_zebrafish$fitIndices[,1], -sign(sft_zebrafish$fitIndices[,3])*sft_zebrafish$fitIndices[,2],
xlab="Soft Threshold (power)",ylab="Scale Free Topology Model Fit,signed R^2",type="n",
main = paste("Scale independence"))
text(sft_zebrafish$fitIndices[,1], -sign(sft_zebrafish$fitIndices[,3])*sft_zebrafish$fitIndices[,2],
labels=powers,cex=cex1,col="blue");
# this line corresponds to using an R^2 cut-off of h
abline(h=0.80,col="blue")
# Mean connectivity as a function of the soft-thresholding power
plot(sft_zebrafish$fitIndices[,1], sft_zebrafish$fitIndices[,5],
xlab="Soft Threshold (power)",ylab="Mean Connectivity", type="n",
main = paste("Mean connectivity"))
text(sft_zebrafish$fitIndices[,1], sft_zebrafish$fitIndices[,5],
labels=powers, cex=cex1,col="blue")
#dev.off()
The soft power chosen will be 14 as this is the minimum one that results in a scale-free network topology R2 value of at least 0.80 for both the zebrafish and human networks.
6.2.2 Constructing the co-expression networks
# 1. Construct adjacency matrices for each dataset, containing each gene's
# connection strength (similar to correlation) to all other genes.
adjacency_human <- adjacency(human_genes, power = 14, type = "signed")
adjacency_zebrafish <- adjacency(zebrafish_genes, power = 14, type = "signed")
#--------------------------------------------------------------------------------------------#
# 2. Calculate whole network connectivity, k, which is defined as the sum of connection
# strengths to other genes for each sample.
connectivity_human <- apply(adjacency_human, 1, sum)
connectivity_zebrafish <- apply(adjacency_zebrafish, 1, sum)
# Scale connectivity to be between 0 and 1.
connectivity_human <- connectivity_human/max(connectivity_human)
connectivity_zebrafish <- connectivity_zebrafish/max(connectivity_zebrafish)
# Plot scale free topology. ------------------------------------------------------------------#
# pdf("data/2017-10-09/scale-free-topology-plots.pdf", width = 8, height = 4)
# par(mfrow=c(1,2), mar=c(4,4,4,4))
# scaleFreePlot(connectivity_human, main="Human")
# scaleFreePlot(connectivity_zebrafish, main="Zebrafish")
# dev.off()
# --------------------------------------------------------------------------------------------#
# --------------------------------------------------------------------------------------------#
# 3. Restrict genes in network to high connectivity genes from either dataset to reduce noise
# during network construction.
# Here, high connectivity is defined as being above the 10th percentile.
minConnections <- 0.1
sufficientlyConnected <- ((connectivity_human > minConnections) | (connectivity_zebrafish > minConnections))
# table(sufficientlyConnected) # genes passing the minimum connectivity threshold.
# Restrict analysis to genes which pass the minimum connectivity threshold.
adjacency_human <- adjacency_human[sufficientlyConnected, sufficientlyConnected]
adjacency_zebrafish <- adjacency_zebrafish[sufficientlyConnected, sufficientlyConnected]
human_genes %<>% extract(,rownames(adjacency_human))
zebrafish_genes %<>% extract(, rownames(adjacency_zebrafish))
geneCounts <- geneCounts[rownames(adjacency_zebrafish),,keep.lib.sizes = FALSE]
# summary(colnames(human_genes)==colnames(zebrafish_genes)) #Check that genes are still in same order for both datasets.
# --------------------------------------------------------------------------------------------#
# 4. Construct topological overlap matrices for the networks.
TOM_human <- TOMsimilarity(adjacency_human, TOMType = "signed", verbose = 0)
colnames(TOM_human) <- rownames(TOM_human) <- rownames(adjacency_human)
TOM_zebrafish <- TOMsimilarity(adjacency_zebrafish, TOMType = "signed", verbose = 0)
colnames(TOM_zebrafish) <- rownames(TOM_zebrafish) <- rownames(adjacency_zebrafish)
# 5. Transform the TOM into dissimilarity matrices.
dissTOM_human <- 1 - TOM_human
dissTOM_zebrafish <- 1 - TOM_zebrafish
# 6. Hierarchial clustering of the dissimilarity TOM:
geneTree_human <- flashClust(as.dist(dissTOM_human), method = "average")
geneTree_zebrafish <- flashClust(as.dist(dissTOM_zebrafish), method = "average")
6.2.3 Module Identification
To identify modules of co-expressed genes, we use a a
# Initialise the color and module eigengene list vectors for the zebrafish dataset.
colorList_zebrafish <- list()
MEList_zebrafish <- list()
#------------------------------------------------------------------------------------#
# METHOD 1: Hybrid Method + PAM
zebrafishHybridCut <- cutreeHybrid(
dendro = geneTree_zebrafish,
distM = dissTOM_zebrafish,
minClusterSize = 40,
deepSplit = 1,
pamStage = TRUE,
maxPamDist = 0.90
)
## ..cutHeight not given, setting it to 0.994 ===> 99% of the (truncated) height range in dendro.
## ..done.
colorList_zebrafish[["HybridCut"]] <- labels2colors(zebrafishHybridCut$labels)
MEList_zebrafish[["HybridCut"]] <- moduleEigengenes(zebrafish_genes, colorList_zebrafish$HybridCut) %>%
use_series("eigengenes") %>%
orderMEs()
colorList_zebrafish$HybridCut %>% table %>% as.data.frame %>%
set_colnames(c("Module", "Genes")) %>% dplyr::arrange(desc(Genes))
## Module Genes
## 1 grey 1553
## 2 turquoise 876
## 3 blue 518
## 4 brown 373
## 5 yellow 318
## 6 green 289
## 7 red 280
## 8 black 258
## 9 pink 257
## 10 magenta 256
## 11 purple 197
## 12 greenyellow 185
## 13 tan 182
## 14 salmon 141
## 15 cyan 138
## 16 midnightblue 138
## 17 lightcyan 135
## 18 grey60 123
## 19 lightgreen 118
## 20 lightyellow 114
## 21 royalblue 112
## 22 darkred 106
## 23 darkgreen 105
## 24 darkgrey 93
## 25 darkturquoise 93
## 26 orange 84
## 27 darkorange 76
## 28 white 75
## 29 skyblue 68
## 30 saddlebrown 59
## 31 steelblue 51
#------------------------------------------------------------------------------------#
# METHOD 2: Hybrid Method + Merge close modules
zebrafishHybridCut_merged <- mergeCloseModules(zebrafish_genes,
colorList_zebrafish$HybridCut,
MEs = MEList_zebrafish$HybridCut,
cutHeight = 0.15)
## mergeCloseModules: Merging modules whose distance is less than 0.15
## Calculating new MEs...
colorList_zebrafish[["HybridCutMerged"]] <- zebrafishHybridCut_merged$colors
MEList_zebrafish[["HybridCutMerged"]] <- zebrafishHybridCut_merged$newMEs
colorList_zebrafish$HybridCutMerged %>% table %>% as.data.frame %>%
set_colnames(c("Module", "Genes")) %>% dplyr::arrange(desc(Genes))
## Module Genes
## 1 grey 1553
## 2 black 1170
## 3 salmon 1017
## 4 brown 373
## 5 lightcyan 332
## 6 yellow 318
## 7 red 280
## 8 pink 257
## 9 magenta 256
## 10 greenyellow 185
## 11 tan 182
## 12 darkred 181
## 13 cyan 138
## 14 midnightblue 138
## 15 grey60 123
## 16 lightgreen 118
## 17 lightyellow 114
## 18 royalblue 112
## 19 darkgrey 93
## 20 darkturquoise 93
## 21 orange 84
## 22 darkorange 76
## 23 skyblue 68
## 24 saddlebrown 59
## 25 steelblue 51
colorList_human <- list()
MEList_human <- list()
#------------------------------------------------------------------------------------#
# METHOD 1: HYBRIDCUT, deepSplit = 2
humanHybridCut <- cutreeHybrid(
dendro = geneTree_human,
distM = dissTOM_human,
minClusterSize = 40,
deepSplit = 2,
pamStage = TRUE,
maxPamDist = 0.90
)
## ..cutHeight not given, setting it to 0.993 ===> 99% of the (truncated) height range in dendro.
## ..done.
colorList_human[["HybridCut"]] <- labels2colors(humanHybridCut$labels)
MEList_human[["HybridCut"]] <- moduleEigengenes(human_genes, colorList_human$HybridCut) %>%
use_series("eigengenes") %>%
orderMEs()
colorList_human$HybridCut %>% table %>% as.data.frame %>%
set_colnames(c("Module", "Genes")) %>% dplyr::arrange(desc(Genes))
## Module Genes
## 1 turquoise 2713
## 2 grey 1146
## 3 blue 884
## 4 brown 798
## 5 yellow 313
## 6 green 301
## 7 red 243
## 8 black 241
## 9 pink 165
## 10 magenta 139
## 11 purple 137
## 12 greenyellow 128
## 13 tan 111
## 14 salmon 52
#------------------------------------------------------------------------------------#
# METHOD 2: HYBRIDCUT, deepSplit = 3
humanHybridCut_2 <- cutreeHybrid(
dendro = geneTree_human,
distM = dissTOM_human,
minClusterSize = 40,
deepSplit = 3,
pamStage = TRUE,
maxPamDist = 0.90
)
## ..cutHeight not given, setting it to 0.993 ===> 99% of the (truncated) height range in dendro.
## ..done.
colorList_human[["HybridCut2"]] <- labels2colors(humanHybridCut_2$labels)
MEList_human[["HybridCut2"]] <- moduleEigengenes(human_genes, colorList_human$HybridCut2) %>%
use_series("eigengenes") %>%
orderMEs()


vim -V filename
run vim with verbose 10 as default and write to messages
show with :messages
vim -V20/tmp/log filename
save verbose 20 to filename /tmp/log
When bigger than zero, Vim will give messages about what it is doing. Currently, these messages are given: >= 1 When the viminfo file is read or written. >= 2 When a file is ":source"'ed. >= 5 Every searched tags file and include file. >= 8 Files for which a group of autocommands is executed. >= 9 Every executed autocommand. >= 12 Every executed function. >= 13 When an exception is thrown, caught, finished, or discarded. >= 14 Anything pending in a ":finally" clause. >= 15 Every executed Ex command (truncated at 200 characters).
-> but no timestamps
set verbosefile=/tmp/verbose.log set verbose=15
only on 7.2.269 or later
vim --startuptime /tmp/profile.log filename
XXX: needs vim to be compiled with +profile
flag
profile start /tmp/log profile func * profile file *
-> written on vim close
vim --cmd 'profile start profile.log' --cmd 'profile func *' --cmd 'profile file *'
notes from help profile
:
only on 7.4
XXX: needs vim to be compiled with +profile
flag
:syntime on
:syntime report
see: http://stackoverflow.com/questions/1687799/profiling-vim-startup-time
vim -V 2> >( ruby -n -e 'puts "#{Time.now.to_f}: #{$_}"' > /tmp/profile.log )
-> only working at the beginning tip is very old (2009) - maybe vim changed since then
-> work around: use fifo -> but vim closes the file -> use while loop
mkfifo ./startuptime vim -V./startuptime while true; do cat startuptime | ruby -n -e 'puts "#{Time.now.to_f}: #{$_}"' >> /tmp/vilog2; done