- destination_template (str) – URL of the target template.
- project_id (str) – Optional, Google Cloud Platform Project ID where the Compute Engine Instance exists. If set to None or missing, the default project_id from the GCP connection is used.
- request_id (str) – Optional, unique request_id that you might add to achieve full idempotence (for example when client call times out repeating the request with the same request id will not create a new instance template again). It should be in UUID format as defined in RFC 4122.
- gcp_conn_id (str) – Optional, The connection ID used to connect to Google Cloud Platform. Defaults to ‘google_cloud_default’.
- api_version (str) – Optional, API version used (for example v1 - or beta). Defaults to v1.
- validate_body (bool) – Optional, If set to False, body validation is not performed. Defaults to False.
Compute Engine Hook¶
-
class
airflow.contrib.hooks.gcp_compute_hook.
GceHook
(api_version='v1', gcp_conn_id='google_cloud_default', delegate_to=None)[source]¶ Bases:
airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
Hook for Google Compute Engine APIs.
All the methods in the hook where project_id is used must be called with keyword arguments rather than positional.
-
get_conn
()[source]¶ Retrieves connection to Google Compute Engine.
Returns: Google Compute Engine services object Return type: dict
-
get_instance_group_manager
(*args, **kwargs)[source]¶ Retrieves Instance Group Manager by project_id, zone and resource_id. Must be called with keyword arguments rather than positional.
Parameters: - zone (str) – Google Cloud Platform zone where the Instance Group Manager exists
- resource_id (str) – Name of the Instance Group Manager
- project_id (str) – Optional, Google Cloud Platform project ID where the Compute Engine Instance exists. If set to None or missing, the default project_id from the GCP connection is used.
Returns: Instance group manager representation as object according to https://cloud.google.com/compute/docs/reference/rest/beta/instanceGroupManagers
Return type: dict
-
get_instance_template
(*args, **kwargs)[source]¶ Retrieves instance template by project_id and resource_id. Must be called with keyword arguments rather than positional.
Parameters: - resource_id (str) – Name of the instance template
- project_id (str) – Optional, Google Cloud Platform project ID where the Compute Engine Instance exists. If set to None or missing, the default project_id from the GCP connection is used.
Returns: Instance template representation as object according to https://cloud.google.com/compute/docs/reference/rest/v1/instanceTemplates
Return type: dict
-
insert_instance_template
(*args, **kwargs)[source]¶ Inserts instance template using body specified Must be called with keyword arguments rather than positional.
Parameters: - body (dict) – Instance template representation as object according to https://cloud.google.com/compute/docs/reference/rest/v1/instanceTemplates
- request_id (str) – Optional, unique request_id that you might add to achieve full idempotence (for example when client call times out repeating the request with the same request id will not create a new instance template again) It should be in UUID format as defined in RFC 4122
- project_id (str) – Optional, Google Cloud Platform project ID where the Compute Engine Instance exists. If set to None or missing, the default project_id from the GCP connection is used.
Returns: None
-
patch_instance_group_manager
(*args, **kwargs)[source]¶ Patches Instance Group Manager with the specified body. Must be called with keyword arguments rather than positional.
Parameters: - zone (str) – Google Cloud Platform zone where the Instance Group Manager exists
- resource_id (str) – Name of the Instance Group Manager
- body (dict) – Instance Group Manager representation as json-merge-patch object according to https://cloud.google.com/compute/docs/reference/rest/beta/instanceTemplates/patch
- request_id (str) – Optional, unique request_id that you might add to achieve full idempotence (for example when client call times out repeating the request with the same request id will not create a new instance template again). It should be in UUID format as defined in RFC 4122
- project_id (str) – Optional, Google Cloud Platform project ID where the Compute Engine Instance exists. If set to None or missing, the default project_id from the GCP connection is used.
:return None
-
set_machine_type
(*args, **kwargs)[source]¶ Sets machine type of an instance defined by project_id, zone and resource_id. Must be called with keyword arguments rather than positional.
Parameters: - zone (str) – Google Cloud Platform zone where the instance exists.
- resource_id (str) – Name of the Compute Engine instance resource
- body (dict) – Body required by the Compute Engine setMachineType API, as described in https://cloud.google.com/compute/docs/reference/rest/v1/instances/setMachineType
- project_id (str) – Optional, Google Cloud Platform project ID where the Compute Engine Instance exists. If set to None or missing, the default project_id from the GCP connection is used.
Returns: None
-
start_instance
(*args, **kwargs)[source]¶ Starts an existing instance defined by project_id, zone and resource_id. Must be called with keyword arguments rather than positional.
Parameters: - zone (str) – Google Cloud Platform zone where the instance exists
- resource_id (str) – Name of the Compute Engine instance resource
- project_id (str) – Optional, Google Cloud Platform project ID where the Compute Engine Instance exists. If set to None or missing, the default project_id from the GCP connection is used.
Returns: None
-
stop_instance
(*args, **kwargs)[source]¶ Stops an instance defined by project_id, zone and resource_id Must be called with keyword arguments rather than positional.
Parameters: - zone (str) – Google Cloud Platform zone where the instance exists
- resource_id (str) – Name of the Compute Engine instance resource
- project_id (str) – Optional, Google Cloud Platform project ID where the Compute Engine Instance exists. If set to None or missing, the default project_id from the GCP connection is used.
Returns: None
-
| members: |
|---|
Cloud Functions¶
Cloud Functions Operators¶
- GcfFunctionDeployOperator : deploy Google Cloud Function to Google Cloud Platform
- GcfFunctionDeleteOperator : delete Google Cloud Function in Google Cloud Platform
They also use Cloud Functions Hook to communicate with Google Cloud Platform.
GcfFunctionDeployOperator¶
-
class
airflow.contrib.operators.gcp_function_operator.
GcfFunctionDeployOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Creates a function in Google Cloud Functions. If a function with this name already exists, it will be updated.
Parameters: - location (str) – Google Cloud Platform region where the function should be created.
- body (dict or google.cloud.functions.v1.CloudFunction) – Body of the Cloud Functions definition. The body must be a Cloud Functions dictionary as described in: https://cloud.google.com/functions/docs/reference/rest/v1/projects.locations.functions . Different API versions require different variants of the Cloud Functions dictionary.
- project_id (str) – (Optional) Google Cloud Platform project ID where the function should be created.
- gcp_conn_id (str) – (Optional) The connection ID used to connect to Google Cloud Platform - default ‘google_cloud_default’.
- api_version (str) – (Optional) API version used (for example v1 - default - or v1beta1).
- zip_path (str) – Path to zip file containing source code of the function. If the path is set, the sourceUploadUrl should not be specified in the body or it should be empty. Then the zip file will be uploaded using the upload URL generated via generateUploadUrl from the Cloud Functions API.
- validate_body (bool) – If set to False, body validation is not performed.
GcfFunctionDeleteOperator¶
-
class
airflow.contrib.operators.gcp_function_operator.
GcfFunctionDeleteOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Deletes the specified function from Google Cloud Functions.
Parameters: - name (str) – A fully-qualified function name, matching the pattern: ^projects/[^/]+/locations/[^/]+/functions/[^/]+$
- gcp_conn_id (str) – The connection ID to use to connect to Google Cloud Platform.
- api_version (str) – API version used (for example v1 or v1beta1).
Cloud Functions Hook¶
-
class
airflow.contrib.hooks.gcp_function_hook.
GcfHook
(api_version, gcp_conn_id='google_cloud_default', delegate_to=None)[source]¶ Bases:
airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
Hook for the Google Cloud Functions APIs.
-
create_new_function
(*args, **kwargs)[source]¶ Creates a new function in Cloud Function in the location specified in the body.
Parameters: - location (str) – The location of the function.
- body (dict) – The body required by the Cloud Functions insert API.
- project_id (str) – Optional, Google Cloud Project project_id where the function belongs. If set to None or missing, the default project_id from the GCP connection is used.
Returns: None
-
delete_function
(name)[source]¶ Deletes the specified Cloud Function.
Parameters: name (str) – The name of the function. Returns: None
-
get_conn
()[source]¶ Retrieves the connection to Cloud Functions.
Returns: Google Cloud Function services object. Return type: dict
-
get_function
(name)[source]¶ Returns the Cloud Function with the given name.
Parameters: name (str) – Name of the function. Returns: A Cloud Functions object representing the function. Return type: dict
-
update_function
(name, body, update_mask)[source]¶ Updates Cloud Functions according to the specified update mask.
Parameters: - name (str) – The name of the function.
- body (dict) – The body required by the cloud function patch API.
- update_mask ([str]) – The update mask - array of fields that should be patched.
Returns: None
-
upload_function_zip
(*args, **kwargs)[source]¶ Uploads zip file with sources.
Parameters: - location (str) – The location where the function is created.
- zip_path (str) – The path of the valid .zip file to upload.
- project_id (str) – Optional, Google Cloud Project project_id where the function belongs. If set to None or missing, the default project_id from the GCP connection is used.
Returns: The upload URL that was returned by generateUploadUrl method.
-
Cloud DataFlow¶
DataFlow Operators¶
- DataFlowJavaOperator : launching Cloud Dataflow jobs written in Java.
- DataflowTemplateOperator : launching a templated Cloud DataFlow batch job.
- DataFlowPythonOperator : launching Cloud Dataflow jobs written in python.
DataFlowJavaOperator¶
-
class
airflow.contrib.operators.dataflow_operator.
DataFlowJavaOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Start a Java Cloud DataFlow batch job. The parameters of the operation will be passed to the job.
See also
For more detail on job submission have a look at the reference: https://cloud.google.com/dataflow/pipelines/specifying-exec-params
Parameters: - jar (string) – The reference to a self executing DataFlow jar.
- dataflow_default_options (dict) – Map of default job options.
- options (dict) – Map of job specific options.
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- poll_sleep (int) – The time in seconds to sleep between polling Google Cloud Platform for the dataflow job status while the job is in the JOB_STATE_RUNNING state.
- job_class (string) – The name of the dataflow job class to be executued, it is often not the main class configured in the dataflow jar file.
Both
jar
andoptions
are templated so you can use variables in them.Note that both
dataflow_default_options
andoptions
will be merged to specify pipeline execution parameter, anddataflow_default_options
is expected to save high-level options, for instances, project and zone information, which apply to all dataflow operators in the DAG.It’s a good practice to define dataflow_* parameters in the default_args of the dag like the project, zone and staging location.
default_args = { 'dataflow_default_options': { 'project': 'my-gcp-project', 'zone': 'europe-west1-d', 'stagingLocation': 'gs://my-staging-bucket/staging/' } }
You need to pass the path to your dataflow as a file reference with the
jar
parameter, the jar needs to be a self executing jar (see documentation here: https://beam.apache.org/documentation/runners/dataflow/#self-executing-jar). Useoptions
to pass on options to your job.t1 = DataFlowJavaOperator( task_id='datapflow_example', jar='{{var.value.gcp_dataflow_base}}pipeline/build/libs/pipeline-example-1.0.jar', options={ 'autoscalingAlgorithm': 'BASIC', 'maxNumWorkers': '50', 'start': '{{ds}}', 'partitionType': 'DAY', 'labels': {'foo' : 'bar'} }, gcp_conn_id='gcp-airflow-service-account', dag=my-dag)
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date':
(2016, 8, 1),
'email'<EMAIL_ADDRESS>'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=30),
'dataflow_default_options': {
'project': 'my-gcp-project',
'zone': 'us-central1-f',
'stagingLocation': 'gs://bucket/tmp/dataflow/staging/',
}
}
dag = DAG('test-dag', default_args=default_args)
task = DataFlowJavaOperator(
gcp_conn_id='gcp_default',
task_id='normalize-cal',
jar='{{var.value.gcp_dataflow_base}}pipeline-ingress-cal-normalize-1.0.jar',
options={
'autoscalingAlgorithm': 'BASIC',
'maxNumWorkers': '50',
'start': '{{ds}}',
'partitionType': 'DAY'
},
dag=dag)
DataflowTemplateOperator¶
-
class
airflow.contrib.operators.dataflow_operator.
DataflowTemplateOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Start a Templated Cloud DataFlow batch job. The parameters of the operation will be passed to the job.
Parameters: - template (string) – The reference to the DataFlow template.
- dataflow_default_options (dict) – Map of default job environment options.
- parameters (dict) – Map of job specific parameters for the template.
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- poll_sleep (int) – The time in seconds to sleep between polling Google Cloud Platform for the dataflow job status while the job is in the JOB_STATE_RUNNING state.
It’s a good practice to define dataflow_* parameters in the default_args of the dag like the project, zone and staging location.
See also
https://cloud.google.com/dataflow/docs/reference/rest/v1b3/LaunchTemplateParameters https://cloud.google.com/dataflow/docs/reference/rest/v1b3/RuntimeEnvironment
default_args = { 'dataflow_default_options': { 'project': 'my-gcp-project' 'zone': 'europe-west1-d', 'tempLocation': 'gs://my-staging-bucket/staging/' } } }
You need to pass the path to your dataflow template as a file reference with the
template
parameter. Useparameters
to pass on parameters to your job. Useenvironment
to pass on runtime environment variables to your job.t1 = DataflowTemplateOperator( task_id='datapflow_example', template='{{var.value.gcp_dataflow_base}}', parameters={ 'inputFile': "gs://bucket/input/my_input.txt", 'outputFile': "gs://bucket/output/my_output.txt" }, gcp_conn_id='gcp-airflow-service-account', dag=my-dag)
template
,dataflow_default_options
andparameters
are templated so you can use variables in them.Note that
dataflow_default_options
is expected to save high-level options for project information, which apply to all dataflow operators in the DAG.See also
https://cloud.google.com/dataflow/docs/reference/rest/v1b3 /LaunchTemplateParameters https://cloud.google.com/dataflow/docs/reference/rest/v1b3/RuntimeEnvironment For more detail on job template execution have a look at the reference: https://cloud.google.com/dataflow/docs/templates/executing-templates
DataFlowPythonOperator¶
-
class
airflow.contrib.operators.dataflow_operator.
DataFlowPythonOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Launching Cloud Dataflow jobs written in python. Note that both dataflow_default_options and options will be merged to specify pipeline execution parameter, and dataflow_default_options is expected to save high-level options, for instances, project and zone information, which apply to all dataflow operators in the DAG.
See also
For more detail on job submission have a look at the reference: https://cloud.google.com/dataflow/pipelines/specifying-exec-params
Parameters: - py_file (string) – Reference to the python dataflow pipleline file.py, e.g., /some/local/file/path/to/your/python/pipeline/file.
- py_options – Additional python options.
- dataflow_default_options (dict) – Map of default job options.
- options (dict) – Map of job specific options.
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- poll_sleep (int) – The time in seconds to sleep between polling Google Cloud Platform for the dataflow job status while the job is in the JOB_STATE_RUNNING state.
Cloud DataProc¶
DataProc Operators¶
- DataprocClusterCreateOperator : Create a new cluster on Google Cloud Dataproc.
- DataprocClusterDeleteOperator : Delete a cluster on Google Cloud Dataproc.
- DataprocClusterScaleOperator : Scale up or down a cluster on Google Cloud Dataproc.
- DataProcPigOperator : Start a Pig query Job on a Cloud DataProc cluster.
- DataProcHiveOperator : Start a Hive query Job on a Cloud DataProc cluster.
- DataProcSparkSqlOperator : Start a Spark SQL query Job on a Cloud DataProc cluster.
- DataProcSparkOperator : Start a Spark Job on a Cloud DataProc cluster.
- DataProcHadoopOperator : Start a Hadoop Job on a Cloud DataProc cluster.
- DataProcPySparkOperator : Start a PySpark Job on a Cloud DataProc cluster.
- DataprocWorkflowTemplateInstantiateOperator : Instantiate a WorkflowTemplate on Google Cloud Dataproc.
- DataprocWorkflowTemplateInstantiateInlineOperator : Instantiate a WorkflowTemplate Inline on Google Cloud Dataproc.
DataprocClusterCreateOperator¶
-
class
airflow.contrib.operators.dataproc_operator.
DataprocClusterCreateOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Create a new cluster on Google Cloud Dataproc. The operator will wait until the creation is successful or an error occurs in the creation process.
The parameters allow to configure the cluster. Please refer to
https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters
for a detailed explanation on the different parameters. Most of the configuration parameters detailed in the link are available as a parameter to this operator.
Parameters: - cluster_name (string) – The name of the DataProc cluster to create. (templated)
- project_id (str) – The ID of the google cloud project in which to create the cluster. (templated)
- num_workers (int) – The # of workers to spin up. If set to zero will spin up cluster in a single node mode
- storage_bucket (string) – The storage bucket to use, setting to None lets dataproc generate a custom one for you
- init_actions_uris (list[string]) – List of GCS uri’s containing dataproc initialization scripts
- init_action_timeout (string) – Amount of time executable scripts in init_actions_uris has to complete
- metadata (dict) – dict of key-value google compute engine metadata entries to add to all instances
- image_version (string) – the version of software inside the Dataproc cluster
- custom_image – custom Dataproc image for more info see https://cloud.google.com/dataproc/docs/guides/dataproc-images
- properties (dict) – dict of properties to set on config files (e.g. spark-defaults.conf), see https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters#SoftwareConfig
- master_machine_type (string) – Compute engine machine type to use for the master node
- master_disk_type (string) – Type of the boot disk for the master node
(default is
pd-standard
). Valid values:pd-ssd
(Persistent Disk Solid State Drive) orpd-standard
(Persistent Disk Hard Disk Drive). - master_disk_size (int) – Disk size for the master node
- worker_machine_type (string) – Compute engine machine type to use for the worker nodes
- worker_disk_type (string) – Type of the boot disk for the worker node
(default is
pd-standard
). Valid values:pd-ssd
(Persistent Disk Solid State Drive) orpd-standard
(Persistent Disk Hard Disk Drive). - worker_disk_size (int) – Disk size for the worker nodes
- num_preemptible_workers (int) – The # of preemptible worker nodes to spin up
- labels (dict) – dict of labels to add to the cluster
- zone (string) – The zone where the cluster will be located. (templated)
- network_uri (string) – The network uri to be used for machine communication, cannot be specified with subnetwork_uri
- subnetwork_uri (string) – The subnetwork uri to be used for machine communication, cannot be specified with network_uri
- internal_ip_only (bool) – If true, all instances in the cluster will only have internal IP addresses. This can only be enabled for subnetwork enabled networks
- tags (list[string]) – The GCE tags to add to all instances
- region – leave as ‘global’, might become relevant in the future. (templated)
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- service_account (string) – The service account of the dataproc instances.
- service_account_scopes (list[string]) – The URIs of service account scopes to be included.
- idle_delete_ttl (int) – The longest duration that cluster would keep alive while staying idle. Passing this threshold will cause cluster to be auto-deleted. A duration in seconds.
- auto_delete_time (datetime.datetime) – The time when cluster will be auto-deleted.
- auto_delete_ttl (int) – The life duration of cluster, the cluster will be auto-deleted at the end of this duration. A duration in seconds. (If auto_delete_time is set this parameter will be ignored)
Type: custom_image: string
DataprocClusterScaleOperator¶
-
class
airflow.contrib.operators.dataproc_operator.
DataprocClusterScaleOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Scale, up or down, a cluster on Google Cloud Dataproc. The operator will wait until the cluster is re-scaled.
Example:
t1 = DataprocClusterScaleOperator( task_id='dataproc_scale', project_id='my-project', cluster_name='cluster-1', num_workers=10, num_preemptible_workers=10, graceful_decommission_timeout='1h', dag=dag)
See also
For more detail on about scaling clusters have a look at the reference: https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/scaling-clusters
Parameters: - cluster_name (string) – The name of the cluster to scale. (templated)
- project_id (string) – The ID of the google cloud project in which the cluster runs. (templated)
- region (string) – The region for the dataproc cluster. (templated)
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- num_workers (int) – The new number of workers
- num_preemptible_workers (int) – The new number of preemptible workers
- graceful_decommission_timeout (string) – Timeout for graceful YARN decomissioning. Maximum value is 1d
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
DataprocClusterDeleteOperator¶
-
class
airflow.contrib.operators.dataproc_operator.
DataprocClusterDeleteOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Delete a cluster on Google Cloud Dataproc. The operator will wait until the cluster is destroyed.
Parameters: - cluster_name (string) – The name of the cluster to create. (templated)
- project_id (string) – The ID of the google cloud project in which the cluster runs. (templated)
- region (string) – leave as ‘global’, might become relevant in the future. (templated)
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
DataProcPigOperator¶
-
class
airflow.contrib.operators.dataproc_operator.
DataProcPigOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Start a Pig query Job on a Cloud DataProc cluster. The parameters of the operation will be passed to the cluster.
It’s a good practice to define dataproc_* parameters in the default_args of the dag like the cluster name and UDFs.
default_args = { 'cluster_name': 'cluster-1', 'dataproc_pig_jars': [ 'gs://example/udf/jar/datafu/1.2.0/datafu.jar', 'gs://example/udf/jar/gpig/1.2/gpig.jar' ] }
You can pass a pig script as string or file reference. Use variables to pass on variables for the pig script to be resolved on the cluster or use the parameters to be resolved in the script as template parameters.
Example:
t1 = DataProcPigOperator( task_id='dataproc_pig', query='a_pig_script.pig', variables={'out': 'gs://example/output/{{ds}}'}, dag=dag)
See also
For more detail on about job submission have a look at the reference: https://cloud.google.com/dataproc/reference/rest/v1/projects.regions.jobs
Parameters: - query (string) – The query or reference to the query file (pg or pig extension). (templated)
- query_uri (string) – The uri of a pig script on Cloud Storage.
- variables (dict) – Map of named parameters for the query. (templated)
- job_name (string) – The job name used in the DataProc cluster. This name by default is the task_id appended with the execution data, but can be templated. The name will always be appended with a random number to avoid name clashes. (templated)
- cluster_name (string) – The name of the DataProc cluster. (templated)
- dataproc_pig_properties (dict) – Map for the Pig properties. Ideal to put in default arguments
- dataproc_pig_jars (list) – URIs to jars provisioned in Cloud Storage (example: for UDFs and libs) and are ideal to put in default arguments.
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- region (str) – The specified region where the dataproc cluster is created.
- job_error_states (list) – Job states that should be considered error states.
Any states in this list will result in an error being raised and failure of the
task. Eg, if the
CANCELLED
state should also be considered a task failure, pass in['ERROR', 'CANCELLED']
. Possible values are currently only'ERROR'
and'CANCELLED'
, but could change in the future. Defaults to['ERROR']
.
Variables: dataproc_job_id (string) – The actual “jobId” as submitted to the Dataproc API. This is useful for identifying or linking to the job in the Google Cloud Console Dataproc UI, as the actual “jobId” submitted to the Dataproc API is appended with an 8 character random string.
DataProcHiveOperator¶
-
class
airflow.contrib.operators.dataproc_operator.
DataProcHiveOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Start a Hive query Job on a Cloud DataProc cluster.
Parameters: - query (string) – The query or reference to the query file (q extension).
- query_uri (string) – The uri of a hive script on Cloud Storage.
- variables (dict) – Map of named parameters for the query.
- job_name (string) – The job name used in the DataProc cluster. This name by default is the task_id appended with the execution data, but can be templated. The name will always be appended with a random number to avoid name clashes.
- cluster_name (string) – The name of the DataProc cluster.
- dataproc_hive_properties (dict) – Map for the Pig properties. Ideal to put in default arguments
- dataproc_hive_jars (list) – URIs to jars provisioned in Cloud Storage (example: for UDFs and libs) and are ideal to put in default arguments.
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- region (str) – The specified region where the dataproc cluster is created.
- job_error_states (list) – Job states that should be considered error states.
Any states in this list will result in an error being raised and failure of the
task. Eg, if the
CANCELLED
state should also be considered a task failure, pass in['ERROR', 'CANCELLED']
. Possible values are currently only'ERROR'
and'CANCELLED'
, but could change in the future. Defaults to['ERROR']
.
Variables: dataproc_job_id (string) – The actual “jobId” as submitted to the Dataproc API. This is useful for identifying or linking to the job in the Google Cloud Console Dataproc UI, as the actual “jobId” submitted to the Dataproc API is appended with an 8 character random string.
DataProcSparkSqlOperator¶
-
class
airflow.contrib.operators.dataproc_operator.
DataProcSparkSqlOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Start a Spark SQL query Job on a Cloud DataProc cluster.
Parameters: - query (string) – The query or reference to the query file (q extension). (templated)
- query_uri (string) – The uri of a spark sql script on Cloud Storage.
- variables (dict) – Map of named parameters for the query. (templated)
- job_name (string) – The job name used in the DataProc cluster. This name by default is the task_id appended with the execution data, but can be templated. The name will always be appended with a random number to avoid name clashes. (templated)
- cluster_name (string) – The name of the DataProc cluster. (templated)
- dataproc_spark_properties (dict) – Map for the Pig properties. Ideal to put in default arguments
- dataproc_spark_jars (list) – URIs to jars provisioned in Cloud Storage (example: for UDFs and libs) and are ideal to put in default arguments.
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- region (str) – The specified region where the dataproc cluster is created.
- job_error_states (list) – Job states that should be considered error states.
Any states in this list will result in an error being raised and failure of the
task. Eg, if the
CANCELLED
state should also be considered a task failure, pass in['ERROR', 'CANCELLED']
. Possible values are currently only'ERROR'
and'CANCELLED'
, but could change in the future. Defaults to['ERROR']
.
Variables: dataproc_job_id (string) – The actual “jobId” as submitted to the Dataproc API. This is useful for identifying or linking to the job in the Google Cloud Console Dataproc UI, as the actual “jobId” submitted to the Dataproc API is appended with an 8 character random string.
DataProcSparkOperator¶
-
class
airflow.contrib.operators.dataproc_operator.
DataProcSparkOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Start a Spark Job on a Cloud DataProc cluster.
Parameters: - main_jar (string) – URI of the job jar provisioned on Cloud Storage. (use this or the main_class, not both together).
- main_class (string) – Name of the job class. (use this or the main_jar, not both together).
- arguments (list) – Arguments for the job. (templated)
- archives (list) – List of archived files that will be unpacked in the work directory. Should be stored in Cloud Storage.
- files (list) – List of files to be copied to the working directory
- job_name (string) – The job name used in the DataProc cluster. This name by default is the task_id appended with the execution data, but can be templated. The name will always be appended with a random number to avoid name clashes. (templated)
- cluster_name (string) – The name of the DataProc cluster. (templated)
- dataproc_spark_properties (dict) – Map for the Pig properties. Ideal to put in default arguments
- dataproc_spark_jars (list) – URIs to jars provisioned in Cloud Storage (example: for UDFs and libs) and are ideal to put in default arguments.
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- region (str) – The specified region where the dataproc cluster is created.
- job_error_states (list) – Job states that should be considered error states.
Any states in this list will result in an error being raised and failure of the
task. Eg, if the
CANCELLED
state should also be considered a task failure, pass in['ERROR', 'CANCELLED']
. Possible values are currently only'ERROR'
and'CANCELLED'
, but could change in the future. Defaults to['ERROR']
.
Variables: dataproc_job_id (string) – The actual “jobId” as submitted to the Dataproc API. This is useful for identifying or linking to the job in the Google Cloud Console Dataproc UI, as the actual “jobId” submitted to the Dataproc API is appended with an 8 character random string.
DataProcHadoopOperator¶
-
class
airflow.contrib.operators.dataproc_operator.
DataProcHadoopOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Start a Hadoop Job on a Cloud DataProc cluster.
Parameters: - main_jar (string) – URI of the job jar provisioned on Cloud Storage. (use this or the main_class, not both together).
- main_class (string) – Name of the job class. (use this or the main_jar, not both together).
- arguments (list) – Arguments for the job. (templated)
- archives (list) – List of archived files that will be unpacked in the work directory. Should be stored in Cloud Storage.
- files (list) – List of files to be copied to the working directory
- job_name (string) – The job name used in the DataProc cluster. This name by default is the task_id appended with the execution data, but can be templated. The name will always be appended with a random number to avoid name clashes. (templated)
- cluster_name (string) – The name of the DataProc cluster. (templated)
- dataproc_hadoop_properties (dict) – Map for the Pig properties. Ideal to put in default arguments
- dataproc_hadoop_jars (list) – URIs to jars provisioned in Cloud Storage (example: for UDFs and libs) and are ideal to put in default arguments.
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- region (str) – The specified region where the dataproc cluster is created.
- job_error_states (list) – Job states that should be considered error states.
Any states in this list will result in an error being raised and failure of the
task. Eg, if the
CANCELLED
state should also be considered a task failure, pass in['ERROR', 'CANCELLED']
. Possible values are currently only'ERROR'
and'CANCELLED'
, but could change in the future. Defaults to['ERROR']
.
Variables: dataproc_job_id (string) – The actual “jobId” as submitted to the Dataproc API. This is useful for identifying or linking to the job in the Google Cloud Console Dataproc UI, as the actual “jobId” submitted to the Dataproc API is appended with an 8 character random string.
DataProcPySparkOperator¶
-
class
airflow.contrib.operators.dataproc_operator.
DataProcPySparkOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Start a PySpark Job on a Cloud DataProc cluster.
Parameters: - main (string) – [Required] The Hadoop Compatible Filesystem (HCFS) URI of the main Python file to use as the driver. Must be a .py file.
- arguments (list) – Arguments for the job. (templated)
- archives (list) – List of archived files that will be unpacked in the work directory. Should be stored in Cloud Storage.
- files (list) – List of files to be copied to the working directory
- pyfiles (list) – List of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip
- job_name (string) – The job name used in the DataProc cluster. This name by default is the task_id appended with the execution data, but can be templated. The name will always be appended with a random number to avoid name clashes. (templated)
- cluster_name (string) – The name of the DataProc cluster.
- dataproc_pyspark_properties (dict) – Map for the Pig properties. Ideal to put in default arguments
- dataproc_pyspark_jars (list) – URIs to jars provisioned in Cloud Storage (example: for UDFs and libs) and are ideal to put in default arguments.
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- region (str) – The specified region where the dataproc cluster is created.
- job_error_states (list) – Job states that should be considered error states.
Any states in this list will result in an error being raised and failure of the
task. Eg, if the
CANCELLED
state should also be considered a task failure, pass in['ERROR', 'CANCELLED']
. Possible values are currently only'ERROR'
and'CANCELLED'
, but could change in the future. Defaults to['ERROR']
.
Variables: dataproc_job_id (string) – The actual “jobId” as submitted to the Dataproc API. This is useful for identifying or linking to the job in the Google Cloud Console Dataproc UI, as the actual “jobId” submitted to the Dataproc API is appended with an 8 character random string.
DataprocWorkflowTemplateInstantiateOperator¶
-
class
airflow.contrib.operators.dataproc_operator.
DataprocWorkflowTemplateInstantiateOperator
(**kwargs)[source]¶ Bases:
airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateBaseOperator
Instantiate a WorkflowTemplate on Google Cloud Dataproc. The operator will wait until the WorkflowTemplate is finished executing.
See also
Please refer to: https://cloud.google.com/dataproc/docs/reference/rest/v1beta2/projects.regions.workflowTemplates/instantiate
Parameters: - template_id (string) – The id of the template. (templated)
- project_id (string) – The ID of the google cloud project in which the template runs
- region (string) – leave as ‘global’, might become relevant in the future
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
DataprocWorkflowTemplateInstantiateInlineOperator¶
-
class
airflow.contrib.operators.dataproc_operator.
DataprocWorkflowTemplateInstantiateInlineOperator
(**kwargs)[source]¶ Bases:
airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateBaseOperator
Instantiate a WorkflowTemplate Inline on Google Cloud Dataproc. The operator will wait until the WorkflowTemplate is finished executing.
See also
Please refer to: https://cloud.google.com/dataproc/docs/reference/rest/v1beta2/projects.regions.workflowTemplates/instantiateInline
Parameters: - template (map) – The template contents. (templated)
- project_id (string) – The ID of the google cloud project in which the template runs
- region (string) – leave as ‘global’, might become relevant in the future
- gcp_conn_id (string) – The connection ID to use connecting to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
Cloud Datastore¶
Datastore Operators¶
- DatastoreExportOperator : Export entities from Google Cloud Datastore to Cloud Storage.
- DatastoreImportOperator : Import entities from Cloud Storage to Google Cloud Datastore.
DatastoreExportOperator¶
-
class
airflow.contrib.operators.datastore_export_operator.
DatastoreExportOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Export entities from Google Cloud Datastore to Cloud Storage
Parameters: - bucket (string) – name of the cloud storage bucket to backup data
- namespace (str) – optional namespace path in the specified Cloud Storage bucket to backup data. If this namespace does not exist in GCS, it will be created.
- datastore_conn_id (string) – the name of the Datastore connection id to use
- cloud_storage_conn_id (string) – the name of the cloud storage connection id to force-write backup
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- entity_filter (dict) – description of what data from the project is included in the export, refer to https://cloud.google.com/datastore/docs/reference/rest/Shared.Types/EntityFilter
- labels (dict) – client-assigned labels for cloud storage
- polling_interval_in_seconds (int) – number of seconds to wait before polling for execution status again
- overwrite_existing (bool) – if the storage bucket + namespace is not empty, it will be emptied prior to exports. This enables overwriting existing backups.
- xcom_push (bool) – push operation name to xcom for reference
DatastoreImportOperator¶
-
class
airflow.contrib.operators.datastore_import_operator.
DatastoreImportOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Import entities from Cloud Storage to Google Cloud Datastore
Parameters: - bucket (string) – container in Cloud Storage to store data
- file (string) – path of the backup metadata file in the specified Cloud Storage bucket. It should have the extension .overall_export_metadata
- namespace (str) – optional namespace of the backup metadata file in the specified Cloud Storage bucket.
- entity_filter (dict) – description of what data from the project is included in the export, refer to https://cloud.google.com/datastore/docs/reference/rest/Shared.Types/EntityFilter
- labels (dict) – client-assigned labels for cloud storage
- datastore_conn_id (string) – the name of the connection id to use
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- polling_interval_in_seconds (int) – number of seconds to wait before polling for execution status again
- xcom_push (bool) – push operation name to xcom for reference
DatastoreHook¶
-
class
airflow.contrib.hooks.datastore_hook.
DatastoreHook
(datastore_conn_id='google_cloud_datastore_default', delegate_to=None)[source]¶ Bases:
airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
Interact with Google Cloud Datastore. This hook uses the Google Cloud Platform connection.
This object is not threads safe. If you want to make multiple requests simultaneously, you will need to create a hook per thread.
-
allocate_ids
(partialKeys)[source]¶ Allocate IDs for incomplete keys. see https://cloud.google.com/datastore/docs/reference/rest/v1/projects/allocateIds
Parameters: partialKeys – a list of partial keys Returns: a list of full keys.
-
commit
(body)[source]¶ Commit a transaction, optionally creating, deleting or modifying some entities.
Parameters: body – the body of the commit request Returns: the response body of the commit request
-
delete_operation
(name)[source]¶ Deletes the long-running operation
Parameters: name – the name of the operation resource
-
export_to_storage_bucket
(bucket, namespace=None, entity_filter=None, labels=None)[source]¶ Export entities from Cloud Datastore to Cloud Storage for backup
-
get_operation
(name)[source]¶ Gets the latest state of a long-running operation
Parameters: name – the name of the operation resource
-
import_from_storage_bucket
(bucket, file, namespace=None, entity_filter=None, labels=None)[source]¶ Import a backup from Cloud Storage to Cloud Datastore
-
lookup
(keys, read_consistency=None, transaction=None)[source]¶ Lookup some entities by key
Parameters: - keys – the keys to lookup
- read_consistency – the read consistency to use. default, strong or eventual. Cannot be used with a transaction.
- transaction – the transaction to use, if any.
Returns: the response body of the lookup request.
-
poll_operation_until_done
(name, polling_interval_in_seconds)[source]¶ Poll backup operation state until it’s completed
-
Cloud ML Engine¶
Cloud ML Engine Operators¶
- MLEngineBatchPredictionOperator : Start a Cloud ML Engine batch prediction job.
- MLEngineModelOperator : Manages a Cloud ML Engine model.
- MLEngineTrainingOperator : Start a Cloud ML Engine training job.
- MLEngineVersionOperator : Manages a Cloud ML Engine model version.
MLEngineBatchPredictionOperator¶
-
class
airflow.contrib.operators.mlengine_operator.
MLEngineBatchPredictionOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Start a Google Cloud ML Engine prediction job.
NOTE: For model origin, users should consider exactly one from the three options below: 1. Populate ‘uri’ field only, which should be a GCS location that points to a tensorflow savedModel directory. 2. Populate ‘model_name’ field only, which refers to an existing model, and the default version of the model will be used. 3. Populate both ‘model_name’ and ‘version_name’ fields, which refers to a specific version of a specific model.
In options 2 and 3, both model and version name should contain the minimal identifier. For instance, call
MLEngineBatchPredictionOperator( ..., model_name='my_model', version_name='my_version', ...)
if the desired model version is “projects/my_project/models/my_model/versions/my_version”.
See https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs for further documentation on the parameters.
Parameters: - project_id (string) – The Google Cloud project name where the prediction job is submitted. (templated)
- job_id (string) – A unique id for the prediction job on Google Cloud ML Engine. (templated)
- data_format (string) – The format of the input data. It will default to ‘DATA_FORMAT_UNSPECIFIED’ if is not provided or is not one of [“TEXT”, “TF_RECORD”, “TF_RECORD_GZIP”].
- input_paths (list of string) – A list of GCS paths of input data for batch prediction. Accepting wildcard operator *, but only at the end. (templated)
- output_path (string) – The GCS path where the prediction results are written to. (templated)
- region (string) – The Google Compute Engine region to run the prediction job in. (templated)
- model_name (string) – The Google Cloud ML Engine model to use for prediction. If version_name is not provided, the default version of this model will be used. Should not be None if version_name is provided. Should be None if uri is provided. (templated)
- version_name (string) – The Google Cloud ML Engine model version to use for prediction. Should be None if uri is provided. (templated)
- uri (string) – The GCS path of the saved model to use for prediction. Should be None if model_name is provided. It should be a GCS path pointing to a tensorflow SavedModel. (templated)
- max_worker_count (int) – The maximum number of workers to be used for parallel processing. Defaults to 10 if not specified.
- runtime_version (string) – The Google Cloud ML Engine runtime version to use for batch prediction.
- gcp_conn_id (string) – The connection ID used for connection to Google Cloud Platform.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have doamin-wide delegation enabled.
- Raises:
ValueError
: if a unique model/version origin cannot be determined.
MLEngineModelOperator¶
-
class
airflow.contrib.operators.mlengine_operator.
MLEngineModelOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Operator for managing a Google Cloud ML Engine model.
Parameters: - project_id (string) – The Google Cloud project name to which MLEngine model belongs. (templated)
- model (dict) –
A dictionary containing the information about the model. If the operation is create, then the model parameter should contain all the information about this model such as name.
If the operation is get, the model parameter should contain the name of the model.
- operation (string) –
The operation to perform. Available operations are:
create
: Creates a new model as provided by the model parameter.get
: Gets a particular model where the name is specified in model.
- gcp_conn_id (string) – The connection ID to use when fetching connection info.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
MLEngineTrainingOperator¶
-
class
airflow.contrib.operators.mlengine_operator.
MLEngineTrainingOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Operator for launching a MLEngine training job.
Parameters: - project_id (string) – The Google Cloud project name within which MLEngine training job should run (templated).
- job_id (string) – A unique templated id for the submitted Google MLEngine training job. (templated)
- package_uris (string) – A list of package locations for MLEngine training job, which should include the main training program + any additional dependencies. (templated)
- training_python_module (string) – The Python module name to run within MLEngine training job after installing ‘package_uris’ packages. (templated)
- training_args (string) – A list of templated command line arguments to pass to the MLEngine training program. (templated)
- region (string) – The Google Compute Engine region to run the MLEngine training job in (templated).
- scale_tier (string) – Resource tier for MLEngine training job. (templated)
- runtime_version (string) – The Google Cloud ML runtime version to use for training. (templated)
- python_version (string) – The version of Python used in training. (templated)
- job_dir (string) – A Google Cloud Storage path in which to store training outputs and other data needed for training. (templated)
- gcp_conn_id (string) – The connection ID to use when fetching connection info.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- mode (string) – Can be one of ‘DRY_RUN’/’CLOUD’. In ‘DRY_RUN’ mode, no real training job will be launched, but the MLEngine training job request will be printed out. In ‘CLOUD’ mode, a real MLEngine training job creation request will be issued.
MLEngineVersionOperator¶
-
class
airflow.contrib.operators.mlengine_operator.
MLEngineVersionOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Operator for managing a Google Cloud ML Engine version.
Parameters: - project_id (string) – The Google Cloud project name to which MLEngine model belongs.
- model_name (string) – The name of the Google Cloud ML Engine model that the version belongs to. (templated)
- version_name (string) – A name to use for the version being operated upon. If not None and the version argument is None or does not have a value for the name key, then this will be populated in the payload for the name key. (templated)
- version (dict) – A dictionary containing the information about the version. If the operation is create, version should contain all the information about this version such as name, and deploymentUrl. If the operation is get or delete, the version parameter should contain the name of the version. If it is None, the only operation possible would be list. (templated)
- operation (string) –
The operation to perform. Available operations are:
create
: Creates a new version in the model specified by model_name, in which case the version parameter should contain all the information to create that version (e.g. name, deploymentUrl).get
: Gets full information of a particular version in the model specified by model_name. The name of the version should be specified in the version parameter.list
: Lists all available versions of the model specified by model_name.delete
: Deletes the version specified in version parameter from the model specified by model_name). The name of the version should be specified in the version parameter.
- gcp_conn_id (string) – The connection ID to use when fetching connection info.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
Cloud ML Engine Hook¶
MLEngineHook¶
-
class
airflow.contrib.hooks.gcp_mlengine_hook.
MLEngineHook
(gcp_conn_id='google_cloud_default', delegate_to=None)[source]¶ Bases:
airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
-
create_job
(project_id, job, use_existing_job_fn=None)[source]¶ Launches a MLEngine job and wait for it to reach a terminal state.
Parameters: - project_id (string) – The Google Cloud project id within which MLEngine job will be launched.
- job (dict) –
MLEngine Job object that should be provided to the MLEngine API, such as:
{ 'jobId': 'my_job_id', 'trainingInput': { 'scaleTier': 'STANDARD_1', ... } }
- use_existing_job_fn (function) – In case that a MLEngine job with the same job_id already exist, this method (if provided) will decide whether we should use this existing job, continue waiting for it to finish and returning the job object. It should accepts a MLEngine job object, and returns a boolean value indicating whether it is OK to reuse the existing job. If ‘use_existing_job_fn’ is not provided, we by default reuse the existing MLEngine job.
Returns: The MLEngine job object if the job successfully reach a terminal state (which might be FAILED or CANCELLED state).
Return type: dict
-
create_version
(project_id, model_name, version_spec)[source]¶ Creates the Version on Google Cloud ML Engine.
Returns the operation if the version was created successfully and raises an error otherwise.
-
delete_version
(project_id, model_name, version_name)[source]¶ Deletes the given version of a model. Blocks until finished.
-
Cloud Storage¶
Storage Operators¶
- FileToGoogleCloudStorageOperator : Uploads a file to Google Cloud Storage.
- GoogleCloudStorageCreateBucketOperator : Creates a new ACL entry on the specified bucket.
- GoogleCloudStorageBucketCreateAclEntryOperator : Creates a new cloud storage bucket.
- GoogleCloudStorageDownloadOperator : Downloads a file from Google Cloud Storage.
- GoogleCloudStorageListOperator : List all objects from the bucket with the give string prefix and delimiter in name.
- GoogleCloudStorageToBigQueryOperator : Creates a new ACL entry on the specified object.
- GoogleCloudStorageObjectCreateAclEntryOperator : Loads files from Google cloud storage into BigQuery.
- GoogleCloudStorageToGoogleCloudStorageOperator : Copies objects from a bucket to another, with renaming if requested.
- GoogleCloudStorageToGoogleCloudStorageTransferOperator : Copies objects from a bucket to another using Google Transfer service.
- MySqlToGoogleCloudStorageOperator: Copy data from any MySQL Database to Google cloud storage in JSON format.
FileToGoogleCloudStorageOperator¶
-
class
airflow.contrib.operators.file_to_gcs.
FileToGoogleCloudStorageOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Uploads a file to Google Cloud Storage. Optionally can compress the file for upload.
Parameters: - src (string) – Path to the local file. (templated)
- dst (string) – Destination path within the specified bucket. (templated)
- bucket (string) – The bucket to upload to. (templated)
- google_cloud_storage_conn_id (string) – The Airflow connection ID to upload with
- mime_type (string) – The mime-type string
- delegate_to (str) – The account to impersonate, if any
- gzip (bool) – Allows for file to be compressed and uploaded as gzip
GoogleCloudStorageBucketCreateAclEntryOperator¶
-
class
airflow.contrib.operators.gcs_acl_operator.
GoogleCloudStorageBucketCreateAclEntryOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Creates a new ACL entry on the specified bucket.
Parameters: - bucket (str) – Name of a bucket.
- entity (str) – The entity holding the permission, in one of the following forms: user-userId, user-email, group-groupId, group-email, domain-domain, project-team-projectId, allUsers, allAuthenticatedUsers
- role (str) – The access permission for the entity. Acceptable values are: “OWNER”, “READER”, “WRITER”.
- user_project (str) – (Optional) The project to be billed for this request. Required for Requester Pays buckets.
- google_cloud_storage_conn_id (str) – The connection ID to use when connecting to Google Cloud Storage.
GoogleCloudStorageCreateBucketOperator¶
-
class
airflow.contrib.operators.gcs_operator.
GoogleCloudStorageCreateBucketOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Creates a new bucket. Google Cloud Storage uses a flat namespace, so you can’t create a bucket with a name that is already in use.
See also
For more information, see Bucket Naming Guidelines: https://cloud.google.com/storage/docs/bucketnaming.html#requirements
Parameters: - bucket_name (string) – The name of the bucket. (templated)
- storage_class (string) –
This defines how objects in the bucket are stored and determines the SLA and the cost of storage (templated). Values include
MULTI_REGIONAL
REGIONAL
STANDARD
NEARLINE
COLDLINE
.
If this value is not specified when the bucket is created, it will default to STANDARD.
- location (string) –
The location of the bucket. (templated) Object data for objects in the bucket resides in physical storage within this region. Defaults to US.
- project_id (string) – The ID of the GCP Project. (templated)
- labels (dict) – User-provided labels, in key/value pairs.
- google_cloud_storage_conn_id (string) – The connection ID to use when connecting to Google cloud storage.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- Example:
The following Operator would create a new bucket
test-bucket
withMULTI_REGIONAL
storage class inEU
regionCreateBucket = GoogleCloudStorageCreateBucketOperator( task_id='CreateNewBucket', bucket_name='test-bucket', storage_class='MULTI_REGIONAL', location='EU', labels={'env': 'dev', 'team': 'airflow'}, google_cloud_storage_conn_id='airflow-service-account' )
GoogleCloudStorageDownloadOperator¶
-
class
airflow.contrib.operators.gcs_download_operator.
GoogleCloudStorageDownloadOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Downloads a file from Google Cloud Storage.
Parameters: - bucket (string) – The Google cloud storage bucket where the object is. (templated)
- object (string) – The name of the object to download in the Google cloud storage bucket. (templated)
- filename (string) – The file path on the local file system (where the operator is being executed) that the file should be downloaded to. (templated) If no filename passed, the downloaded data will not be stored on the local file system.
- store_to_xcom_key (string) – If this param is set, the operator will push the contents of the downloaded file to XCom with the key set in this parameter. If not set, the downloaded data will not be pushed to XCom. (templated)
- google_cloud_storage_conn_id (string) – The connection ID to use when connecting to Google cloud storage.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
GoogleCloudStorageListOperator¶
-
class
airflow.contrib.operators.gcs_list_operator.
GoogleCloudStorageListOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
List all objects from the bucket with the give string prefix and delimiter in name.
- This operator returns a python list with the name of objects which can be used by
- xcom in the downstream task.
Parameters: - bucket (string) – The Google cloud storage bucket to find the objects. (templated)
- prefix (string) – Prefix string which filters objects whose name begin with this prefix. (templated)
- delimiter (string) – The delimiter by which you want to filter the objects. (templated) For e.g to lists the CSV files from in a directory in GCS you would use delimiter=’.csv’.
- google_cloud_storage_conn_id (string) – The connection ID to use when connecting to Google cloud storage.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- Example:
The following Operator would list all the Avro files from
sales/sales-2017
folder indata
bucket.GCS_Files = GoogleCloudStorageListOperator( task_id='GCS_Files', bucket='data', prefix='sales/sales-2017/', delimiter='.avro', google_cloud_storage_conn_id=google_cloud_conn_id )
GoogleCloudStorageObjectCreateAclEntryOperator¶
-
class
airflow.contrib.operators.gcs_acl_operator.
GoogleCloudStorageObjectCreateAclEntryOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Creates a new ACL entry on the specified object.
Parameters: - bucket (str) – Name of a bucket.
- object_name (str) – Name of the object. For information about how to URL encode object names to be path safe, see: https://cloud.google.com/storage/docs/json_api/#encoding
- entity (str) – The entity holding the permission, in one of the following forms: user-userId, user-email, group-groupId, group-email, domain-domain, project-team-projectId, allUsers, allAuthenticatedUsers
- role (str) – The access permission for the entity. Acceptable values are: “OWNER”, “READER”.
- generation (str) – (Optional) If present, selects a specific revision of this object (as opposed to the latest version, the default).
- user_project (str) – (Optional) The project to be billed for this request. Required for Requester Pays buckets.
- google_cloud_storage_conn_id (str) – The connection ID to use when connecting to Google Cloud Storage.
GoogleCloudStorageToBigQueryOperator¶
-
class
airflow.contrib.operators.gcs_to_bq.
GoogleCloudStorageToBigQueryOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Loads files from Google cloud storage into BigQuery.
The schema to be used for the BigQuery table may be specified in one of two ways. You may either directly pass the schema fields in, or you may point the operator to a Google cloud storage object name. The object in Google cloud storage must be a JSON file with the schema fields in it.
Parameters: - bucket (string) – The bucket to load from. (templated)
- source_objects (list of str) – List of Google cloud storage URIs to load from. (templated) If source_format is ‘DATASTORE_BACKUP’, the list must only contain a single URI.
- destination_project_dataset_table (string) – The dotted (<project>.)<dataset>.<table> BigQuery table to load data into. If <project> is not included, project will be the project defined in the connection json. (templated)
- schema_fields (list) – If set, the schema field list as defined here: https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load Should not be set when source_format is ‘DATASTORE_BACKUP’.
- schema_object (string) – If set, a GCS object path pointing to a .json file that contains the schema for the table. (templated)
- source_format (string) – File format to export.
- compression (string) – [Optional] The compression type of the data source. Possible values include GZIP and NONE. The default value is NONE. This setting is ignored for Google Cloud Bigtable, Google Cloud Datastore backups and Avro formats.
- create_disposition (string) – The create disposition if the table doesn’t exist.
- skip_leading_rows (int) – Number of rows to skip when loading from a CSV.
- write_disposition (string) – The write disposition if the table already exists.
- field_delimiter (string) – The delimiter to use when loading from a CSV.
- max_bad_records (int) – The maximum number of bad records that BigQuery can ignore when running the job.
- quote_character (string) – The value that is used to quote data sections in a CSV file.
- ignore_unknown_values (bool) – [Optional] Indicates if BigQuery should allow extra values that are not represented in the table schema. If true, the extra values are ignored. If false, records with extra columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result.
- allow_quoted_newlines (bool) – Whether to allow quoted newlines (true) or not (false).
- allow_jagged_rows (bool) – Accept rows that are missing trailing optional columns. The missing values are treated as nulls. If false, records with missing trailing columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result. Only applicable to CSV, ignored for other formats.
- max_id_key (string) – If set, the name of a column in the BigQuery table that’s to be loaded. This will be used to select the MAX value from BigQuery after the load occurs. The results will be returned by the execute() command, which in turn gets stored in XCom for future operators to use. This can be helpful with incremental loads–during future executions, you can pick up from the max ID.
- bigquery_conn_id (string) – Reference to a specific BigQuery hook.
- google_cloud_storage_conn_id (string) – Reference to a specific Google cloud storage hook.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- schema_update_options (list) – Allows the schema of the destination table to be updated as a side effect of the load job.
- src_fmt_configs (dict) – configure optional fields specific to the source format
- external_table (bool) – Flag to specify if the destination table should be a BigQuery external table. Default Value is False.
- time_partitioning (dict) – configure optional time partitioning fields i.e. partition by field, type and expiration as per API specifications. Note that ‘field’ is not available in concurrency with dataset.table$partition.
- cluster_fields (list of str) – Request that the result of this load be stored sorted by one or more columns. This is only available in conjunction with time_partitioning. The order of columns given determines the sort order. Not applicable for external tables.
GoogleCloudStorageToGoogleCloudStorageOperator¶
-
class
airflow.contrib.operators.gcs_to_gcs.
GoogleCloudStorageToGoogleCloudStorageOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Copies objects from a bucket to another, with renaming if requested.
Parameters: - source_bucket (string) – The source Google cloud storage bucket where the object is. (templated)
- source_object (string) –
The source name of the object to copy in the Google cloud storage bucket. (templated) If wildcards are used in this argument:
You can use only one wildcard for objects (filenames) within your bucket. The wildcard can appear inside the object name or at the end of the object name. Appending a wildcard to the bucket name is unsupported. - destination_bucket (string) – The destination Google cloud storage bucket where the object should be. (templated)
- destination_object (string) – The destination name of the object in the
destination Google cloud storage bucket. (templated)
If a wildcard is supplied in the source_object argument, this is the
prefix that will be prepended to the final destination objects’ paths.
Note that the source path’s part before the wildcard will be removed;
if it needs to be retained it should be appended to destination_object.
For example, with prefix
foo/*
and destination_objectblah/
, the filefoo/baz
will be copied toblah/baz
; to retain the prefix write the destination_object as e.g.blah/foo
, in which case the copied file will be namedblah/foo/baz
. - move_object (bool) – When move object is True, the object is moved instead of copied to the new location. This is the equivalent of a mv command as opposed to a cp command.
- google_cloud_storage_conn_id (string) – The connection ID to use when connecting to Google cloud storage.
- delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- Examples:
The following Operator would copy a single file named
sales/sales-2017/january.avro
in thedata
bucket to the file namedcopied_sales/2017/january-backup.avro` in the ``data_backup
bucketcopy_single_file = GoogleCloudStorageToGoogleCloudStorageOperator( task_id='copy_single_file', source_bucket='data', source_object='sales/sales-2017/january.avro', destination_bucket='data_backup', destination_object='copied_sales/2017/january-backup.avro', google_cloud_storage_conn_id=google_cloud_conn_id )
The following Operator would copy all the Avro files from
sales/sales-2017
folder (i.e. with names starting with that prefix) indata
bucket to thecopied_sales/2017
folder in thedata_backup
bucket.copy_files = GoogleCloudStorageToGoogleCloudStorageOperator( task_id='copy_files', source_bucket='data', source_object='sales/sales-2017/*.avro', destination_bucket='data_backup', destination_object='copied_sales/2017/', google_cloud_storage_conn_id=google_cloud_conn_id )
The following Operator would move all the Avro files from
sales/sales-2017
folder (i.e. with names starting with that prefix) indata
bucket to the same folder in thedata_backup
bucket, deleting the original files in the process.move_files = GoogleCloudStorageToGoogleCloudStorageOperator( task_id='move_files', source_bucket='data', source_object='sales/sales-2017/*.avro', destination_bucket='data_backup', move_object=True, google_cloud_storage_conn_id=google_cloud_conn_id )
GoogleCloudStorageToGoogleCloudStorageTransferOperator¶
-
class
airflow.contrib.operators.gcs_to_gcs_transfer_operator.
GoogleCloudStorageToGoogleCloudStorageTransferOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Copies objects from a bucket to another using the GCP Storage Transfer Service.
Parameters: - source_bucket (str) – The source Google cloud storage bucket where the object is. (templated)
- destination_bucket (str) – The destination Google cloud storage bucket where the object should be. (templated)
- project_id (str) – The ID of the Google Cloud Platform Console project that owns the job
- gcp_conn_id (str) – Optional connection ID to use when connecting to Google Cloud Storage.
- delegate_to (str) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
- description (str) – Optional transfer service job description
- schedule (dict) – Optional transfer service schedule; see https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs. If not set, run transfer job once as soon as the operator runs
- object_conditions (dict) – Optional transfer service object conditions; see https://cloud.google.com/storage-transfer/docs/reference/rest/v1/TransferSpec#ObjectConditions
- transfer_options (dict) – Optional transfer service transfer options; see https://cloud.google.com/storage-transfer/docs/reference/rest/v1/TransferSpec#TransferOptions
- wait (bool) – Wait for transfer to finish; defaults to True
Example:
gcs_to_gcs_transfer_op = GoogleCloudStorageToGoogleCloudStorageTransferOperator( task_id='gcs_to_gcs_transfer_example', source_bucket='my-source-bucket', destination_bucket='my-destination-bucket', project_id='my-gcp-project', dag=my_dag)
MySqlToGoogleCloudStorageOperator¶
-
class
airflow.contrib.operators.mysql_to_gcs.
MySqlToGoogleCloudStorageOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Copy data from MySQL to Google cloud storage in JSON format.
GoogleCloudStorageHook¶
-
class
airflow.contrib.hooks.gcs_hook.
GoogleCloudStorageHook
(google_cloud_storage_conn_id='google_cloud_default', delegate_to=None)[source]¶ Bases:
airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
Interact with Google Cloud Storage. This hook uses the Google Cloud Platform connection.
-
copy
(source_bucket, source_object, destination_bucket=None, destination_object=None)[source]¶ Copies an object from a bucket to another, with renaming if requested.
destination_bucket or destination_object can be omitted, in which case source bucket/object is used, but not both.
Parameters: - source_bucket (string) – The bucket of the object to copy from.
- source_object (string) – The object to copy.
- destination_bucket (string) – The destination of the object to copied to. Can be omitted; then the same bucket is used.
- destination_object (string) – The (renamed) path of the object if given. Can be omitted; then the same name is used.
-
create_bucket
(bucket_name, storage_class='MULTI_REGIONAL', location='US', project_id=None, labels=None)[source]¶ Creates a new bucket. Google Cloud Storage uses a flat namespace, so you can’t create a bucket with a name that is already in use.
See also
For more information, see Bucket Naming Guidelines: https://cloud.google.com/storage/docs/bucketnaming.html#requirements
Parameters: - bucket_name (string) – The name of the bucket.
- storage_class (string) –
This defines how objects in the bucket are stored and determines the SLA and the cost of storage. Values include
MULTI_REGIONAL
REGIONAL
STANDARD
NEARLINE
COLDLINE
.
If this value is not specified when the bucket is created, it will default to STANDARD.
- location (string) –
The location of the bucket. Object data for objects in the bucket resides in physical storage within this region. Defaults to US.
- project_id (string) – The ID of the GCP Project.
- labels (dict) – User-provided labels, in key/value pairs.
Returns: If successful, it returns the
id
of the bucket.
-
delete
(bucket, object, generation=None)[source]¶ Delete an object if versioning is not enabled for the bucket, or if generation parameter is used.
Parameters: - bucket (string) – name of the bucket, where the object resides
- object (string) – name of the object to delete
- generation (string) – if present, permanently delete the object of this generation
Returns: True if succeeded
-
download
(bucket, object, filename=None)[source]¶ Get a file from Google Cloud Storage.
Parameters: - bucket (string) – The bucket to fetch from.
- object (string) – The object to fetch.
- filename (string) – If set, a local file path where the file should be written to.
-
exists
(bucket, object)[source]¶ Checks for the existence of a file in Google Cloud Storage.
Parameters: - bucket (string) – The Google cloud storage bucket where the object is.
- object (string) – The name of the object to check in the Google cloud storage bucket.
-
get_crc32c
(bucket, object)[source]¶ Gets the CRC32c checksum of an object in Google Cloud Storage.
Parameters: - bucket (string) – The Google cloud storage bucket where the object is.
- object (string) – The name of the object to check in the Google cloud storage bucket.
-
get_md5hash
(bucket, object)[source]¶ Gets the MD5 hash of an object in Google Cloud Storage.
Parameters: - bucket (string) – The Google cloud storage bucket where the object is.
- object (string) – The name of the object to check in the Google cloud storage bucket.
-
get_size
(bucket, object)[source]¶ Gets the size of a file in Google Cloud Storage.
Parameters: - bucket (string) – The Google cloud storage bucket where the object is.
- object (string) – The name of the object to check in the Google cloud storage bucket.
-
insert_bucket_acl
(bucket, entity, role, user_project)[source]¶ Creates a new ACL entry on the specified bucket. See: https://cloud.google.com/storage/docs/json_api/v1/bucketAccessControls/insert
Parameters: - bucket (str) – Name of a bucket.
- entity (str) – The entity holding the permission, in one of the following forms: user-userId, user-email, group-groupId, group-email, domain-domain, project-team-projectId, allUsers, allAuthenticatedUsers. See: https://cloud.google.com/storage/docs/access-control/lists#scopes
- role (str) – The access permission for the entity. Acceptable values are: “OWNER”, “READER”, “WRITER”.
- user_project (str) – (Optional) The project to be billed for this request. Required for Requester Pays buckets.
-
insert_object_acl
(bucket, object_name, entity, role, generation, user_project)[source]¶ Creates a new ACL entry on the specified object. See: https://cloud.google.com/storage/docs/json_api/v1/objectAccessControls/insert
Parameters: - bucket (str) – Name of a bucket.
- object_name (str) – Name of the object. For information about how to URL encode object names to be path safe, see: https://cloud.google.com/storage/docs/json_api/#encoding
- entity (str) – The entity holding the permission, in one of the following forms: user-userId, user-email, group-groupId, group-email, domain-domain, project-team-projectId, allUsers, allAuthenticatedUsers See: https://cloud.google.com/storage/docs/access-control/lists#scopes
- role (str) – The access permission for the entity. Acceptable values are: “OWNER”, “READER”.
- generation (str) – (Optional) If present, selects a specific revision of this object (as opposed to the latest version, the default).
- user_project (str) – (Optional) The project to be billed for this request. Required for Requester Pays buckets.
-
is_updated_after
(bucket, object, ts)[source]¶ Checks if an object is updated in Google Cloud Storage.
Parameters: - bucket (string) – The Google cloud storage bucket where the object is.
- object (string) – The name of the object to check in the Google cloud storage bucket.
- ts (datetime) – The timestamp to check against.
-
list
(bucket, versions=None, maxResults=None, prefix=None, delimiter=None)[source]¶ List all objects from the bucket with the give string prefix in name
Parameters: - bucket (string) – bucket name
- versions (boolean) – if true, list all versions of the objects
- maxResults (integer) – max count of items to return in a single page of responses
- prefix (string) – prefix string which filters objects whose name begin with this prefix
- delimiter (string) – filters objects based on the delimiter (for e.g ‘.csv’)
Returns: a stream of object names matching the filtering criteria
-
rewrite
(source_bucket, source_object, destination_bucket, destination_object=None)[source]¶ Has the same functionality as copy, except that will work on files over 5 TB, as well as when copying between locations and/or storage classes.
destination_object can be omitted, in which case source_object is used.
Parameters: - source_bucket (string) – The bucket of the object to copy from.
- source_object (string) – The object to copy.
- destination_bucket (string) – The destination of the object to copied to.
- destination_object – The (renamed) path of the object if given. Can be omitted; then the same name is used.
-
upload
(bucket, object, filename, mime_type='application/octet-stream', gzip=False, multipart=False, num_retries=0)[source]¶ Uploads a local file to Google Cloud Storage.
Parameters: - bucket (string) – The bucket to upload to.
- object (string) – The object name to set when uploading the local file.
- filename (string) – The local file path to the file to be uploaded.
- mime_type (str) – The MIME type to set when uploading the file.
- gzip (bool) – Option to compress file for upload
- multipart (bool or int) – If True, the upload will be split into multiple HTTP requests. The default size is 256MiB per request. Pass a number instead of True to specify the request size, which must be a multiple of 262144 (256KiB).
- num_retries (int) – The number of times to attempt to re-upload the file (or individual chunks, in the case of multipart uploads). Retries are attempted with exponential backoff.
-
GCPTransferServiceHook¶
-
class
airflow.contrib.hooks.gcp_transfer_hook.
GCPTransferServiceHook
(api_version='v1', gcp_conn_id='google_cloud_default', delegate_to=None)[source]¶ Bases:
airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
Hook for GCP Storage Transfer Service.
Google Kubernetes Engine¶
Google Kubernetes Engine Cluster Operators¶
- GKEClusterDeleteOperator : Creates a Kubernetes Cluster in Google Cloud Platform
- GKEPodOperator : Deletes a Kubernetes Cluster in Google Cloud Platform
GKEClusterCreateOperator¶
GKEClusterDeleteOperator¶
GKEPodOperator¶
Google Kubernetes Engine Hook¶
Qubole¶
Apache Airflow has a native operator and hooks to talk to Qubole, which lets you submit your big data jobs directly to Qubole from Apache Airflow.
QuboleOperator¶
-
class
airflow.contrib.operators.qubole_operator.
QuboleOperator
(**kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Execute tasks (commands) on QDS (https://qubole.com).
Parameters: qubole_conn_id (str) – Connection id which consists of qds auth_token - kwargs:
command_type: type of command to be executed, e.g. hivecmd, shellcmd, hadoopcmd tags: array of tags to be assigned with the command cluster_label: cluster label on which the command will be executed name: name to be given to command notify: whether to send email on command completion or not (default is False) Arguments specific to command types
- hivecmd:
query: inline query statement script_location: s3 location containing query statement sample_size: size of sample in bytes on which to run query macros: macro values which were used in query - prestocmd:
query: inline query statement script_location: s3 location containing query statement macros: macro values which were used in query - hadoopcmd:
sub_commnad: must be one these [“jar”, “s3distcp”, “streaming”] followed by 1 or more args - shellcmd:
script: inline command with args script_location: s3 location containing query statement files: list of files in s3 bucket as file1,file2 format. These files will be copied into the working directory where the qubole command is being executed. archives: list of archives in s3 bucket as archive1,archive2 format. These will be unarchived intothe working directory where the qubole command is being executed parameters: any extra args which need to be passed to script (only when script_location is supplied) - pigcmd:
script: inline query statement (latin_statements) script_location: s3 location containing pig query parameters: any extra args which need to be passed to script (only when script_location is supplied - sparkcmd:
program: the complete Spark Program in Scala, SQL, Command, R, or Python cmdline: spark-submit command line, all required information must be specify in cmdline itself. sql: inline sql query script_location: s3 location containing query statement language: language of the program, Scala, SQL, Command, R, or Python app_id: ID of an Spark job server app arguments: spark-submit command line arguments user_program_arguments: arguments that the user program takes in macros: macro values which were used in query - dbtapquerycmd:
db_tap_id: data store ID of the target database, in Qubole. query: inline query statement macros: macro values which were used in query - dbexportcmd:
mode: 1 (simple), 2 (advance) hive_table: Name of the hive table partition_spec: partition specification for Hive table. dbtap_id: data store ID of the target database, in Qubole. db_table: name of the db table db_update_mode: allowinsert or updateonly db_update_keys: columns used to determine the uniqueness of rows export_dir: HDFS/S3 location from which data will be exported. fields_terminated_by: hex of the char used as column separator in the dataset - dbimportcmd:
mode: 1 (simple), 2 (advance) hive_table: Name of the hive table dbtap_id: data store ID of the target database, in Qubole. db_table: name of the db table where_clause: where clause, if any parallelism: number of parallel db connections to use for extracting data extract_query: SQL query to extract data from db. $CONDITIONS must be part of the where clause. boundary_query: Query to be used get range of row IDs to be extracted split_column: Column used as row ID to split data into ranges (mode 2)
Note
Following fields are template-supported :
query
,script_location
,sub_command
,script
,files
,archives
,program
,cmdline
,sql
,where_clause
,extract_query
,boundary_query
,macros
,tags
,name
,parameters
,dbtap_id
,hive_table
,db_table
,split_column
,note_id
,db_update_keys
,export_dir
,partition_spec
,qubole_conn_id
,arguments
,user_program_arguments
.You can also use.txt
files for template driven use cases.Note
In QuboleOperator there is a default handler for task failures and retries, which generally kills the command running at QDS for the corresponding task instance. You can override this behavior by providing your own failure and retry handler in task definition.
QubolePartitionSensor¶
-
class
airflow.contrib.sensors.qubole_sensor.
QubolePartitionSensor
(**kwargs)[source]¶ Bases:
airflow.contrib.sensors.qubole_sensor.QuboleSensor
Wait for a Hive partition to show up in QHS (Qubole Hive Service) and check for its presence via QDS APIs
Parameters: - qubole_conn_id (str) – Connection id which consists of qds auth_token
- data (a JSON object) – a JSON object containing payload, whose presence needs to be checked. Check this example for sample payload structure.
Note
Both
data
andqubole_conn_id
fields support templating. You can also use.txt
files for template-driven use cases.
QuboleFileSensor¶
-
class
airflow.contrib.sensors.qubole_sensor.
QuboleFileSensor
(**kwargs)[source]¶ Bases:
airflow.contrib.sensors.qubole_sensor.QuboleSensor
Wait for a file or folder to be present in cloud storage and check for its presence via QDS APIs
Parameters: - qubole_conn_id (str) – Connection id which consists of qds auth_token
- data (a JSON object) –
a JSON object containing payload, whose presence needs to be checked Check this example for sample payload structure.
Note
Both
data
andqubole_conn_id
fields support templating. You can also use.txt
files for template-driven use cases.
QuboleCheckOperator¶
-
class
airflow.contrib.operators.qubole_check_operator.
QuboleCheckOperator
(**kwargs)[source]¶ Bases:
airflow.operators.check_operator.CheckOperator
,airflow.contrib.operators.qubole_operator.QuboleOperator
Performs checks against Qubole Commands.
QuboleCheckOperator
expects a command that will be executed on QDS. By default, each value on first row of the result of this Qubole Command is evaluated using pythonbool
casting. If any of the values returnFalse
, the check is failed and errors out.Note that Python bool casting evals the following as
False
:False
0
- Empty string (
""
) - Empty list (
[]
) - Empty dictionary or set (
{}
)
Given a query like
SELECT COUNT(*) FROM foo
, it will fail only if the count== 0
. You can craft much more complex query that could, for instance, check that the table has the same number of rows as the source table upstream, or that the count of today’s partition is greater than yesterday’s partition, or that a set of metrics are less than 3 standard deviation for the 7 day average.This operator can be used as a data quality check in your pipeline, and depending on where you put it in your DAG, you have the choice to stop the critical path, preventing from publishing dubious data, or on the side and receive email alerts without stopping the progress of the DAG.
Parameters: qubole_conn_id (str) – Connection id which consists of qds auth_token kwargs:
Arguments specific to Qubole command can be referred from QuboleOperator docs.
results_parser_callable: This is an optional parameter to extend the flexibility of parsing the results of Qubole command to the users. This is a python callable which can hold the logic to parse list of rows returned by Qubole command. By default, only the values on first row are used for performing checks. This callable should return a list of records on which the checks have to be performed. Note
All fields in common with template fields of QuboleOperator and CheckOperator are template-supported.
QuboleValueCheckOperator¶
-
class
airflow.contrib.operators.qubole_check_operator.
QuboleValueCheckOperator
(**kwargs)[source]¶ Bases:
airflow.operators.check_operator.ValueCheckOperator
,airflow.contrib.operators.qubole_operator.QuboleOperator
Performs a simple value check using Qubole command. By default, each value on the first row of this Qubole command is compared with a pre-defined value. The check fails and errors out if the output of the command is not within the permissible limit of expected value.
Parameters: - qubole_conn_id (str) – Connection id which consists of qds auth_token
- pass_value (str/int/float) – Expected value of the query results.
- tolerance (int/float) – Defines the permissible pass_value range, for example if tolerance is 2, the Qubole command output can be anything between -2*pass_value and 2*pass_value, without the operator erring out.
kwargs:
Arguments specific to Qubole command can be referred from QuboleOperator docs.
results_parser_callable: This is an optional parameter to extend the flexibility of parsing the results of Qubole command to the users. This is a python callable which can hold the logic to parse list of rows returned by Qubole command. By default, only the values on first row are used for performing checks. This callable should return a list of records on which the checks have to be performed. Note
All fields in common with template fields of QuboleOperator and ValueCheckOperator are template-supported.


Snappers!
HOME
ABOUT
USER
CONTACT
UPLOAD
Profile
Logout
Username
Email
Old Password
Confirm Password
New Password
Update


Toggle navigation
Start
Home
Home
About this website
Imprint/Impressum
dissemination services
listPlace to GeoJSON
CMFI to GEXF
Visualize CMI as Network
Visualize CMI as Timeline
listPlace to Map
TEI to HTML
CMDI to HTML
OWL to HTML
Transform2TEI
Make your choice
Data endpoints
Text as XML


AIIMS PG Result 2019 to be Announced Today at aiimsexams.org | All You Need to Know Here
The AIIMS entrance exam was conducted in a computer-based mode on November 17, 2019.
AIIMS PG Result 2019: The All India Institute of Medical Sciences will on Friday announce the results of AIIMS PG entrance exam for various courses namely MD, MS, MDS, 6 yrs DM, 6yrs MCh on its official website aiimsexams.org.
Students are advised to keep a track on the aforementioned website in case there is an update on the same.
“Final results for all courses will only be available on AIIMS website at www.aiimsexams.org. Result of individual candidate will NOT be informed on telephone and candidates are requested NOT to make call to the Examination Section for such information,” the official AIIMS PG entrance notification read.
Once released on the official website, candidates will be able to download the PDF format result of AIIMS PG 2019 entrance test for the January 2020 session.
The AIIMS entrance exam was conducted in a computer-based mode on November 17, 2019.
Candidates will also be able to check their AIIMS PG 2019 seat matrix for the various courses online at the official website. It must be noted that few changes have been made in the seating position for the various PG programmes across AIIMS campuses.
Meanwhile, the National Board of Education closed the NEET PG 2020 registrations on Thursday, i.e., November 21.
For breaking news and live news updates, like us on Facebook or follow us on Twitter and Instagram. Read more on Exams & Results Latest News on India.com.
Story Timeline
- 21 Nov 2019 AIIMS PG Entrance Exam 2020: Results to be Announced Tomorrow at aiimsexams.org
- 11 Oct 2019 AIIMS PG Admission Exam 2020: Registration Date Extended to October 16, Apply on aiimsexam.org
- 21 Feb 2017 AIIMS PG 2017: Open counselling for vacant seats to be held at New Delhi at aiimsexams.org
- 12 Feb 2017 AIIMS PG MD/MS: Important Instruction to register for Open Counseling
Comments - Join the Discussion


This JMX operation requires no arguments. Click the 'Execute' button to invoke the operation.
This JMX operation requires some parameters. Fill in the fields below and click the 'Execute' button to invoke the operation.
Result:
{{$ctrl.operationResult}}