Amazon Data Pipeline · Arazzo Workflow

Amazon Data Pipeline Redeploy Definition

Version 1.0.0

Deactivate a pipeline, write a new definition, then reactivate it with the new objects.

1 workflow 1 source API 1 provider
View Spec View on GitHub Data ProcessingETLWorkflowsData PipelineAutomationArazzoWorkflows

Provider

amazon-data-pipeline

Workflows

redeploy-pipeline-definition
Deactivate, replace the definition, and reactivate a pipeline.
Chains deactivatePipeline, putPipelineDefinition, and activatePipeline so a new definition is applied to an existing pipeline without leaving it running against a stale or partially written definition.
3 steps inputs: cancelActive, pipelineId, pipelineObjects, startTimestamp outputs: errored, pipelineId, reactivated
1
deactivatePipeline
deactivatePipeline
Stop the pipeline's current schedule before its definition is rewritten.
2
putDefinition
putPipelineDefinition
Commit the new pipeline objects to the now-deactivated pipeline.
3
reactivatePipeline
activatePipeline
Reactivate the pipeline so it resumes processing with the updated definition.

Source API Descriptions

Arazzo Workflow Specification

amazon-data-pipeline-redeploy-definition-workflow.yml Raw ↑
arazzo: 1.0.1
info:
  title: Amazon Data Pipeline Redeploy Definition
  summary: Deactivate a pipeline, write a new definition, then reactivate it with the new objects.
  description: >-
    A controlled redeploy flow for AWS Data Pipeline. A running pipeline is
    deactivated so its current schedule stops, the new set of pipeline objects is
    committed with PutPipelineDefinition, and the pipeline is then reactivated to
    resume processing with the updated definition. Every step spells out its
    request inline so the flow can be read and executed without opening the
    underlying OpenAPI description.
  version: 1.0.0
sourceDescriptions:
- name: dataPipelineApi
  url: ../openapi/amazon-data-pipeline-openapi.yml
  type: openapi
workflows:
- workflowId: redeploy-pipeline-definition
  summary: Deactivate, replace the definition, and reactivate a pipeline.
  description: >-
    Chains deactivatePipeline, putPipelineDefinition, and activatePipeline so a
    new definition is applied to an existing pipeline without leaving it running
    against a stale or partially written definition.
  inputs:
    type: object
    required:
    - pipelineId
    - pipelineObjects
    properties:
      pipelineId:
        type: string
        description: The id of the pipeline being redeployed.
      pipelineObjects:
        type: array
        description: The new pipeline objects to commit before reactivation.
        items:
          type: object
      cancelActive:
        type: boolean
        description: Whether to cancel running activities during the deactivation step.
      startTimestamp:
        type: string
        description: The optional date-time at which the reactivated pipeline should begin processing.
  steps:
  - stepId: deactivatePipeline
    description: >-
      Stop the pipeline's current schedule before its definition is rewritten.
    operationId: deactivatePipeline
    requestBody:
      contentType: application/json
      payload:
        pipelineId: $inputs.pipelineId
        cancelActive: $inputs.cancelActive
    successCriteria:
    - condition: $statusCode == 200
    outputs:
      deactivated: $statusCode
  - stepId: putDefinition
    description: >-
      Commit the new pipeline objects to the now-deactivated pipeline.
    operationId: putPipelineDefinition
    requestBody:
      contentType: application/json
      payload:
        pipelineId: $inputs.pipelineId
        pipelineObjects: $inputs.pipelineObjects
    successCriteria:
    - condition: $statusCode == 200
    outputs:
      errored: $response.body#/errored
      validationErrors: $response.body#/validationErrors
  - stepId: reactivatePipeline
    description: >-
      Reactivate the pipeline so it resumes processing with the updated
      definition.
    operationId: activatePipeline
    requestBody:
      contentType: application/json
      payload:
        pipelineId: $inputs.pipelineId
        startTimestamp: $inputs.startTimestamp
    successCriteria:
    - condition: $statusCode == 200
    outputs:
      reactivated: $statusCode
  outputs:
    pipelineId: $inputs.pipelineId
    errored: $steps.putDefinition.outputs.errored
    reactivated: $steps.reactivatePipeline.outputs.reactivated