Orchestrate Batch Loop: batch size, custom breaks, Group By, batch numbers and missing items

How Batch Loop splits a dataset into batches

Batch Loop takes an iterator, cuts it into batches and runs the steps inside the loop once per batch rather than once per item. You choose how it cuts with the Strategy property: Size (a fixed number of items per batch), Custom (a new batch starts at or after an item that matches an expression) or Group (items that share a Group By value land in the same batch).

The component is available in the Synchronous, Asynchronous, Business Process, Home Card, Integration System and Custom Agent Action templates, according to the Batch Loop reference. Besides the strategy you set a Reference Name, a Data Type for the iterator (or AutoType), a Data Set expression and an optional Filter.

In the Orchestrate Sampler app from Workday's App Catalog, the Data Set is an XPath iterator over repeating elements:

data.SampleXML.message.iterator("/Workers/Worker")

Each Worker element is one item. The batch itself is assembled from those items: in the sample's source, each batch is an XML fold with <Batch> as its header and </Batch> as its footer, and the steps inside the loop read it as data.BatchLoop.batch. The per-item value, data.BatchLoop.item, appears only in the expressions that decide where items go: the custom break condition, the Group By expression and the fold that builds the batch.

Size strategy: a fixed number of items per batch

Set Strategy to Size and give Data Breakpoint Number an expression that returns the batch size. The last batch holds whatever is left over. The sampler's size example batches 11 workers by 3, which gives three full batches and a final batch of 2.

Size is the strategy for "send at most N records per call". The Orchestrate Sampler's import example uses a size of 1000 and gives each batch a SOAP request envelope as its header and footer, so every batch is already a complete import request when the loop body sees it:

header: <wd:Import_Applicant_Request wd:version="v45.0" xmlns:wd="urn:com.workday/bsvc">
footer: </wd:Import_Applicant_Request>

Inside the loop a Trigger Integration step passes data.ImportApplicantsInBatchesOf1000.batch to a child integration as its document, and an aggregation collects the import ID each child returns. A Loop after the Batch Loop then checks each import's status. The same shape works for most bulk loads: batch, hand each batch over, collect a handle per batch, then verify by handle.

How big should a batch be? The docs give no recommended number. Choose it from the receiving side: the request size the target accepts and how long one call takes. The orchestration's own limits frame the choice: the runtime limits page caps an orchestration at 200 MB of concurrent data in memory and gives an Integration System orchestration 48 hours to run, with a 500 MB ceiling on any single HTTP response. The looping and pagination limits guide covers the volume side in more detail.

Custom strategy: break at or after a matching item

Set Strategy to Custom, choose At or After in Form Batch, and write a boolean expression that identifies the item where a batch boundary falls. At starts a new batch with the matching item. After starts the new batch with the item that follows the match.

The sampler uses this condition, which matches a worker whose Amount is 200:

data.BatchAtNoKeep.item.numberAtXPath("/Worker/Amount").equals(200)

The checkbox that matters most is Keep Leading Items. The Batch Loop reference shows it with a seven-item list [1,2,3,4,5,6,7] and a break at 5:

With the box clear, everything before the first match (and, with After, the match itself) is dropped. No error is raised, so a run with this box clear simply reports fewer items than the source held.

Custom batching suits files whose structure marks the boundaries, such as a header record that starts each group of detail records: break At the header and keep leading items only if records before the first header mean something.

Group strategy: one batch per Group By value

Set Strategy to Group, write a Group By expression that returns a Number or a String, and give the group a Group Name. Each distinct value becomes a batch, and downstream steps read the current value through that name. The sampler groups workers by email address:

Group By:   data.BatchLoop.item.stringAtXPath("/Worker/emailID")
Group Name: EmailGroup

Inside the loop, a Create Values step totals the batch and a text template writes one line per group:

data.BatchLoop.batch.numbersAtXPath("//Amount").sum()

<Worker>
  <Email>{{data.BatchLoop.EmailGroup}}</Email>
  <SumAmount>{{data.Calculate.sum}}</SumAmount>
</Worker>

Further grouping rules (Add Group By) apply in order: the first rule makes groups, the second splits each of those groups, and so on. Each rule can sort its groups ascending or descending.

Sort By works differently from Group By, and the reference is explicit: Orchestrate sorts the items inside each batch, not the whole dataset. Text sorts lexicographically and numbers numerically, so a numeric ID stored as a string sorts "10" before "9". Convert it to a number in the Sort By expression if order matters. String grouping and sorting depend on locale; the Locale field overrides the app's Default Locale for this loop only.

How to number batches

The Batch Loop reference documents no batch counter. Use what each strategy already gives you.

With the Group strategy, the Group Name value is the batch's identity, and it is usually a better file name or log key than a sequence number. With the Size or Custom strategy, take the identity from the data, for example the first and last employee ID inside the batch, read from data.BatchLoop.batch with the same XPath or JSONPath functions you use elsewhere.

If you need a running number, the plain Loop gives you one. Workday's sample apps read itemNumber on a Loop step, for example data.PhotoLoop.itemNumber.toString().leftPad(4, "0") in a log message. Aggregate each batch into a list in the Batch Loop (a JSON Fold, XML Fold or String List output), then run a Loop over that list and number the batches there. Log the first iteration's value once to see where the count starts before you put it in a file name.

Why the Batch Loop seems to skip items

Work through these in order. Each one is visible in the loop's configuration or in a single logged value.

1. Keep Leading Items is clear on a Custom strategy. Items before the first match are discarded, as in the table above. Select the box, or make the first record always match the condition.
2. The Filter removed them. The Filter narrows the Data Set, so filtered items never reach a batch. Log the count with and without it.
3. The Data Set selects the wrong level. iterator("/Workers") yields one item, the whole document, so you get one batch whatever the strategy. Point the XPath or JSONPath at the repeating element.
4. The aggregation's Condition excluded them. Every aggregation strategy has a Condition, and only items for which it evaluates to true are aggregated, according to Add Aggregations. On a CSV aggregation, Workday's Orchestrate course manual describes a Fail When No Inputs option for when an empty result should stop the run rather than produce an empty file.
5. The body was written for one record per pass. Each pass of a Batch Loop receives a whole batch, and in Workday's samples the steps inside the loop work on data.<LoopName>.batch. A step that reads a single value from it, such as stringAtXPath, returns one value for the whole batch, not one per record. Loop over the batch with a nested Loop when each record needs its own step.
6. A Group By value is empty or inconsistent. Items with different spellings or case of the same key form separate batches. Normalise the key in the Group By expression.

Prove the batching before you send anything

Before the loop body does anything that changes a system, give the Batch Loop an aggregation with a Count output and a second output that collects each batch (an XML Fold with a header and footer, as the sampler does). After the loop, write the folded output to a file with Store Document and attach it to the integration event, then compare the count with the number of items in the source.

When the two agree and every batch in the file holds what you expect, add the step that sends each batch. For a bulk load through Trigger Integration, keep the handle each child returns in an aggregation, as the import sample does, and read every child's result back before treating the run as done. Trigger Integration's retry settings cover failure to trigger the child, not failure of the child's own run. Trigger Integration exists only in the Integration System template and can't be used inside a Join Loop.

How the loop body runs across batches (in parallel or one after another) is a separate question, covered in whether Orchestrate loops run in parallel. To match records from two sources before batching them, use a Join Loop first.