Google Cloud Dataflow · Arazzo Workflow

Google Cloud Dataflow Cleanup Job Snapshots

Version 1.0.0

List snapshots for a job, inspect the oldest one, then delete it.

1 workflow 1 source API 1 provider
View Spec View on GitHub Apache BeamBatch ProcessingBig DataData ProcessingETLStream ProcessingArazzoWorkflows

Provider

google-cloud-dataflow

Workflows

cleanup-job-snapshots
List a job's snapshots, inspect the first, and delete it.
Lists snapshots created from a job, captures the first snapshot id, reads it to confirm its state, then deletes it to reclaim storage.
3 steps inputs: accessToken, jobId, location, projectId outputs: deletedSnapshotId, deletedSnapshotState
1
listSnapshots
listLocationSnapshots
List the snapshots in the region that were created from the supplied source job and capture the first snapshot id.
2
inspectSnapshot
getLocationSnapshot
Read the first snapshot to confirm its state and creation time before deleting it.
3
deleteSnapshot
deleteLocationSnapshot
Permanently delete the inspected snapshot and its associated data to reclaim storage.

Source API Descriptions

Arazzo Workflow Specification

google-cloud-dataflow-cleanup-job-snapshots-workflow.yml Raw ↑
arazzo: 1.0.1
info:
  title: Google Cloud Dataflow Cleanup Job Snapshots
  summary: List snapshots for a job, inspect the oldest one, then delete it.
  description: >-
    Reclaims storage by pruning snapshots created from a specific job. The
    workflow lists the snapshots in a region scoped to a source job, captures the
    first snapshot id, reads that snapshot to confirm its state and creation
    time, then deletes it. Every step spells out its request inline, including
    the inline Bearer authorization Google Cloud requires, so the flow can be
    read and executed without opening the underlying OpenAPI description.
  version: 1.0.0
sourceDescriptions:
- name: dataflowApi
  url: ../openapi/google-cloud-dataflow-api-openapi.yml
  type: openapi
workflows:
- workflowId: cleanup-job-snapshots
  summary: List a job's snapshots, inspect the first, and delete it.
  description: >-
    Lists snapshots created from a job, captures the first snapshot id, reads it
    to confirm its state, then deletes it to reclaim storage.
  inputs:
    type: object
    required:
    - accessToken
    - projectId
    - location
    - jobId
    properties:
      accessToken:
        type: string
        description: Google Cloud OAuth 2.0 access token used as a Bearer credential.
      projectId:
        type: string
        description: The Google Cloud project id that owns the snapshots.
      location:
        type: string
        description: The regional endpoint that contains the snapshots (e.g. us-central1).
      jobId:
        type: string
        description: The source job id whose snapshots should be listed and pruned.
  steps:
  - stepId: listSnapshots
    description: >-
      List the snapshots in the region that were created from the supplied
      source job and capture the first snapshot id.
    operationId: listLocationSnapshots
    parameters:
    - name: Authorization
      in: header
      value: Bearer $inputs.accessToken
    - name: projectId
      in: path
      value: $inputs.projectId
    - name: location
      in: path
      value: $inputs.location
    - name: jobId
      in: query
      value: $inputs.jobId
    successCriteria:
    - condition: $statusCode == 200
    outputs:
      firstSnapshotId: $response.body#/snapshots/0/id
  - stepId: inspectSnapshot
    description: >-
      Read the first snapshot to confirm its state and creation time before
      deleting it.
    operationId: getLocationSnapshot
    parameters:
    - name: Authorization
      in: header
      value: Bearer $inputs.accessToken
    - name: projectId
      in: path
      value: $inputs.projectId
    - name: location
      in: path
      value: $inputs.location
    - name: snapshotId
      in: path
      value: $steps.listSnapshots.outputs.firstSnapshotId
    successCriteria:
    - condition: $statusCode == 200
    outputs:
      snapshotId: $response.body#/id
      state: $response.body#/state
      creationTime: $response.body#/creationTime
  - stepId: deleteSnapshot
    description: >-
      Permanently delete the inspected snapshot and its associated data to
      reclaim storage.
    operationId: deleteLocationSnapshot
    parameters:
    - name: Authorization
      in: header
      value: Bearer $inputs.accessToken
    - name: projectId
      in: path
      value: $inputs.projectId
    - name: location
      in: path
      value: $inputs.location
    - name: snapshotId
      in: path
      value: $steps.inspectSnapshot.outputs.snapshotId
    successCriteria:
    - condition: $statusCode == 200
    outputs:
      deletedSnapshotId: $steps.inspectSnapshot.outputs.snapshotId
  outputs:
    deletedSnapshotId: $steps.deleteSnapshot.outputs.deletedSnapshotId
    deletedSnapshotState: $steps.inspectSnapshot.outputs.state