Workday Orchestrate Looping, Pagination, and Batching Over Large Datasets

Workday Orchestrate handles large datasets through dedicated paged request steps (Send Paged Workday REST Call, Send Paged Workday SOAP Call, and Send Paged HTTP Request) paired with Loop, Batch Loop, and Join Loop components. To process tens of thousands of records without failing, you must run heavy workloads in Asynchronous or Integration System templates, chunk payloads into smaller batches, and let intermediate variables fall out of scope before hitting the 200 MB concurrent memory ceiling or 60-minute production process timeout.

Know the Orchestrate runtime and memory limits first

Before designing a loop or pagination flow, match your expected volume against Workday's documented runtime, process, and memory thresholds. Choosing a synchronous template for a dataset that takes longer than a few seconds is the most common cause of mid-flight orchestration failures.

Paginating Workday REST, WQL, SOAP, and external HTTP calls

Rather than building manual offset counters with recursive self-calls, use Orchestrate's built-in paged request steps whenever an endpoint returns collections.

1. Send Paged Workday REST Call (and WQL pagination)

Standard Workday REST GET collection endpoints default to limit=20 with a maximum of 100 (or 1000 for specific REST APIs and nested prompt endpoints). When querying via Workday Query Language (WQL), a query builds a result cache of up to 1,000,000 total rows (provided you include a LIMIT clause when the underlying dataset exceeds 1 million rows) and returns up to 10,000 rows per page via limit and offset. Note that WQL caches paginated requests only for the duration of a user session, up to 30 minutes, and requests through the Workday Cloud Platform API Gateway time out after 5 minutes (compared to 30 minutes on a tenanted host).

In Send Paged Workday REST Call, configure the General tab:

• Limit: Number of items per page (for example, 100 for standard REST resources or up to 10000 for WQL /data queries).
• Offset: Starting index (set to 0 to start at the first record).
• Max Pages: Maximum pages to retrieve. Leaving this blank retrieves all available pages.
• Flatten Result: When set to true, Orchestrate merges the items across all retrieved pages into a single dataset rather than emitting an array of page envelopes.

Be careful with Flatten Result on massive datasets: flattening hundreds of pages into a single variable keeps the entire combined collection in scope at once. If you need to transform and discard each page's records incrementally, keep Flatten Result set to false and iterate over the page responses.

2. Send Paged Workday SOAP Call

Workday SOAP Web Services return XML responses paged via Response_Filter (Page and Count). The Send Paged Workday SOAP Call step handles iterating over the paged XML responses automatically.

• On the General tab, choose your Credential (Default Workday API Credential or Access Token from Initiating User), enter the Workday web Service name (such as Human_Resources), and specify the Version. Set Include Invalid Reference Responses to true if you need Invalid_Reference_ID_Response elements exposed for error logging.
• On the Body tab, reference existing XML data or build the request payload using Create Text Template. Do not include <soapenv:Envelope> in your template; Orchestrate wraps your body in the SOAP envelope automatically, and adding a duplicate envelope invalidates the request.

3. Send Paged HTTP Request for external APIs

External SaaS APIs rarely share the same pagination contract. Send Paged HTTP Request (which uses GET exclusively) supports two primary modes on its Pagination tab:

• Offset Pagination: Define Limit, Limit Parameter Name (such as $.limit or $.per_page), Initial Offset, Offset Parameter Name (such as $.offset or $.skip), and the JSON Data Path pointing to the results array.
• Cursor Pagination: Choose one of three cursor mechanisms:
  • Cursor token in response body: Map Next Page Parameter Name (for example, $.cursor or $.starting_after) and Next Page Token Path (for example, $.cursor or $.marker).
  • Next page URI in response body: Provide the Next Page URI Path (such as $.links_next) and toggle URI Type to indicate whether the returned URI is absolute or a relative path appended to the base URL.
  • Next page URI in HTTP Link header: Specify the Link Relationship Name (such as next) adhering to RFC 5988.

When using Next Page URIs, never duplicate query parameters on the Query Parameters tab that are already embedded inside the returned Next Page URI, or Orchestrate will construct malformed URLs on page 2 and beyond.

Choosing the right Loop step: Loop, Batch Loop, or Join Loop

Orchestrate provides three distinct looping components. Picking the right one prevents nested lookup calls and keeps downstream payload sizes manageable.

Using Batch Loop as a splitter

When sending records to an external API bulk endpoint or a suborchestration, sending 50,000 individual HTTP requests inside a standard Loop will exhaust connection pools and trigger rate limits. Instead, place a Batch Loop step upstream:

1. Select the Size strategy and set Data Breakpoint Number (for example, 200).
2. Or select the Group strategy and set Group By to a string or numeric expression (such as cost center or location ID) so each batch represents one logical group. Note that when you add Sort By rules inside a Batch Loop, Orchestrate sorts items within each individual batch rather than sorting the entire dataset globally.

Replacing N+1 lookups with Join Loop

If you have 10,000 worker records from a primary extract and need to attach department metadata from a second dataset, do not put a REST or SOAP call inside a 10,000-iteration Loop. Fetch both datasets first and join them in a Join Loop:

• Put the dataset with unusually large records as the Primary Dataset (which drives the loop) and the smaller lookup collection as the Joining Dataset (capped at 500 MB).
• Configure Multiple Match Strategy (Complete with Error, Skip Record, Use First, Use Last, or Use All) and Missing Match Strategy (Alternate Processing using isDataDefined, Complete with Error, or Skip Record).

Stopping loops early and parallel vs. sequential execution

By default, Orchestrate processes loop iterations in parallel to maximize throughput. If you only need to process items until a specific condition is met:

1. Open the related actions menu on the Loop step and select Create Early Stop Condition.
2. Define the boolean Early Stop Condition expression. When triggered, the loop outputs did.Stop.Early = true.
3. Only check Force Sequential Processing when strict ordering is mandatory for your stop condition. Workday explicitly warns that forcing sequential processing significantly increases runtime duration and can trigger process timeouts on large datasets.

Troubleshooting orchestration timeouts and memory errors

• Presentation page call fails at 25 seconds: Any synchronous orchestration launched from a PMD page has a hard 25-second ceiling that overrides orchestration settings. Refactor long-running loops into an Asynchronous orchestration and return a job status or notification instead.
• Process times out at 45 or 60 minutes: Individual processes cap at 45 minutes in non-production tenants and 60 minutes in production. Check if Force Sequential Processing was enabled on a Loop, or split the workload using Batch Loop and suborchestrations.
• Concurrent memory (200 MB) or disk (2 GB) exceeded: Keep large intermediate variables inside child scopes (such as inside a loop block or suborchestration) rather than holding multiple flattened datasets at the root level. Once a block finishes and its values go out of scope, the runtime reclaims both memory and file-backed disk storage.

How kiweely fits

Designing paged WQL queries, calculating batch sizes, and verifying expression paths across nested Batch Loop and Join Loop blocks takes careful coordination. Kiweely inspects your tenant schema, tests WQL pagination queries directly against your environment, and generates Workday Extend orchestration definitions that respect Workday's runtime and memory limits from the start. Kiweely Personal is free for your Workday personal developer account and its tenant. Any other tenant needs a Personal licence, $99 a month, or a seat in your firm.

Frequently asked questions

Why does my WQL pagination query fail in Orchestrate when the dataset has over 1 million rows?

WQL queries return a maximum of 1,000,000 rows. If the underlying query matches more than 1 million rows and does not include a LIMIT clause (set to <= 1000000), the query fails immediately. Add a LIMIT clause or narrow your WHERE filter, and paginate up to 10,000 rows at a time using the limit and offset query parameters before the 30-minute WQL session cache expires.

What is the difference between LIMIT and limit in a WQL REST request?

Although the keyword is not case-sensitive, Workday uses uppercase LIMIT inside the WQL query string to cap the total number of rows the query can return, and lowercase limit as the URL query parameter to control how many rows (up to 10,000) are returned per page alongside offset.

Should I enable Force Sequential Processing on a Loop over thousands of records?

No, unless your business logic strictly depends on each item finishing before the next begins. Orchestrate executes loop iterations in parallel by default; enabling Force Sequential Processing on a Loop with an Early Stop Condition serializes execution and is a frequent cause of hitting the 45-minute (non-production) or 60-minute (production) process timeout.