AWS Bites Episode 141: Step Functions with JSONata and Variables

Date: March 21, 2025
Hosts: Eoin Shanaghy (“Owen”) and Luciano Mammino

Overview

In this episode, Eoin and Luciano dive into two transformative new features for AWS Step Functions: JSONata support and the introduction of variables. They discuss how these additions are making workflows more powerful and developer-friendly, breaking down what these features are, how to use them, and the real-world impact on serverless architectures. The conversation provides practical examples, highlights benefits and limitations, and offers opinions on the overall developer experience.

1. Step Functions Recap

(00:52–06:20)

What Step Functions Are

• Step Functions allow developers to orchestrate distributed workflows without writing glue code.
• Declarative model: define workflows (state machines) with things like steps, conditions, parallelization, and error handling.
• Use cases: ecommerce order flows, ETL processes, data transformations, and automating tasks such as podcast transcription (as used for AWS Bites).

Quote:
“Step Functions allow you to avoid writing code in order to define some kind of workflow or state machine... it can be, I don't know, an ecommerce order flow... or maybe an ETL or some kind of other data transformation process.”
(B, 00:57)

Benefits

• Managed service: No OS or dependency headaches.
• Robust error handling: retries, archives, redrives.
• Observability: view each step’s input/output and pinpoint where errors happen.
• Deep AWS integration: direct links with many AWS services and SDK integration.

Drawbacks

• Local development limitations: Testing locally isn’t great; solutions like LocalStack exist but aren’t always perfect.
• Syntax complexity: JSON/YAML can be limiting for complex scenarios; often leads to a reliance on Lambda functions for data transformations.
• State size limits: 256 KB state data cap was a known pain point.

2. Introduction to JSONata in Step Functions

(06:36–11:24)

What is JSONata?

• JSONata is a sophisticated JSON query and transformation language (heavily used by IBM, inspired by XPath).
• Supports:
  • String and numerical ops
  • Date/time conversions
  • Regular expressions, comparisons, conditionals
  • Array and object manipulations (sorting, grouping, aggregation)
  • Custom functions, map/filter/reduce
• Far more powerful and user-friendly than previous JSONPath support.

Quote:
“It is a JSON query language which was inspired by XPath from the world of XML and it allows you to create sophisticated queries to transform and extract data from JSON.”
(A, 06:38)

Using JSONata in Step Functions

• Flexible adoption: Set query language at workflow or individual state level.
• Replace confusing fields like InputPath, OutputPath, etc., with clear JSONata expressions for output and arguments.
• Reference state input with $State.Input.

Quote:
“You just specify a JSONata expression for either the output field… or you can specify JSONata for an arguments property as well.”
(A, 08:27)

• Tooling: Online playgrounds (official and by Steady) help, but IDE integration is still developing.
• CDK support: Not fully native yet; must use custom states with raw ASL for now.

Developer Experience

• Simpler, more readable workflows without awkward JSONPath syntax.
• Easier onboarding and migration from previous setups (useful guide and video by Eric Johnson linked in show notes).

Quote:
“Overall, the experience with JSONata… is a simpler, easier to read, easier to understand… a much more developer friendly experience and way more powerful.”
(A, 10:55)

3. Step Functions Variables

(11:26–14:07)

What Are Variables?

• Addresses the 256 KB state limitation, which previously led to awkward workarounds (e.g., storing data in S3/DynamoDB and using Lambda to fetch it).
• Variables let you declare named values in one state and access them later, without threading them through every transition.
• New limit: Up to 10 MB per workflow.

Quote:
“Now there's a new feature called variables and that allows you to declare variables in one state and then reference them in any subsequent state without having to propagate them all the way down the state chain.”
(A, 11:33)

Assigning and Using Variables

• Assign via JSONata/JSONPath (often with JSONata for advanced needs).
• Great for:
  • API integrations (e.g., handling bearer tokens, filtering large responses, fanning out actions).
  • Data transformation (reducing the need for multiple Lambda functions).

Quote:
“We were able to just remove a whole lot of lambda functions, which has great benefit. You know, you reduce your deployment time, you reduce your maintenance, you reduce the runtimes you need to keep track of, and it just simplifies everything.”
(A, 13:23)

4. Pricing and Drawbacks

(14:08–15:22)

Cost

• No extra charges for using JSONata or Variables.
• Standard Step Functions: pay per transition (now possibly lower if you combine steps with JSONata).
• Express: pay by execution time.

Quote:
“No, there's no additional cost for using variables or jsonata. I would definitely recommend people give them a try and let us know what you think.”
(A, 14:24)

Caveats

• Learning curve: Slight, but better than VTL (Velocity Template Language).
• Tooling: Ecosystem is maturing—best support still via online playgrounds.

5. Closing Thoughts

(15:22-end)

• Eoin and Luciano encourage listeners to share their experiences and workflows using Step Functions, JSONata, and variables.
• These features mark a significant step forward for AWS workflow automation.
• Real-world impact: simplified architectures, fewer Lambda functions, clearer state handling, and potentially lower costs.

Notable Quotes & Moments

• “The main thing is that you don't have to use programming language... It's effectively a declarative approach.”
  (B, 01:32)

• “You also don't have that dollar syntax for property names that you might keep forgetting anymore either... just makes your whole workflow very easy to understand by reading it.”
  (A, 10:51 / 13:44)

Key Segment Timestamps

• Recap of Step Functions: 00:52–06:20
• JSONata Introduction: 06:36–11:24
• Variables Feature: 11:26–14:07
• Cost & Drawbacks Discussion: 14:08–15:22

Summary:
This episode offers a comprehensive look at two new features in AWS Step Functions—JSONata integration and variables. Both make complex serverless workflows easier to manage, more expressive, and cost-effective, with direct impact on real-world applications. The hosts share best practices, practical examples, migration tips, and set realistic expectations for anyone looking to upgrade their Step Functions usage.