Sometimes we need to run operations that will take a long time. We need to make sure the request doesn’t time out and it would be nice to let the user see that progress is being made. Drupal uses batch processing to do things like rebuilding content access permissions. Let’s dig in and see how we can create a custom batch process of our own. This is a walkthrough on batch processing for Drupal 8. I'm using Drupal 8.4, but this should apply to Drupal 8.2 and up.
The first step is creating a new module. We’ll call ours example_batch.
Here’s your info file (modules/custom/example_batch/example_batch.info.yml):
name: 'Example Batch Processing'
description: 'Provides an example of how to do custom batch processing.'
type: module
core: 8.x
version: 8.x-1.0-dev
package: BT Examples
We’re also going to want a form where we can kick off our batch.
modules/custom/example_batch/example_batch.routing.yml:
example_batch.form:
path: '/example_batch'
defaults:
_form: '\Drupal\example_batch\Form\KickOffBatchForm'
_title: 'Batch Example'
requirements:
_role: 'authenticated'
And our form:
modules/custom/example_batch/src/Form/KickOffBatchForm.php:
<?php
namespace Drupal\example_batch\Form;
use Drupal\Core\Form\FormBase;
use Drupal\Core\Form\FormStateInterface;
/**
* Class KickOffBatchForm.
*/
class KickOffBatchForm extends FormBase {
/**
* {@inheritdoc}
*/
public function getFormId() {
return 'kick_off_batch_form';
}
/**
* {@inheritdoc}
*/
public function buildForm(array $form, FormStateInterface $form_state) {
$form['number_to_execute'] = [
'#type' =----> 'number',
'#title' => $this->t('Number of times to run'),
'#required' => TRUE,
'#min' => 1,
];
$form['actions'] = ['#type' => 'actions'];
$form['actions']['submit'] = [
'#type' => 'submit',
'#value' => t('Execute'),
];
return $form;
}
/**
* {@inheritdoc}
*/
public function validateForm(array &$form, FormStateInterface $form_state) {
parent::validateForm($form, $form_state);
}
/**
* {@inheritdoc}
*/
public function submitForm(array &$form, FormStateInterface $form_state) {
$number = $form_state->getValue('number_to_execute');
$batch = [
'init_message' => t('Executing a batch...'),
'operations' => [
[
'_awesome_batch',
[$number],
],
],
'file' => drupal_get_path('module', 'example_batch') . '/example_batch.awesome_batch.inc',
];
batch_set($batch);
}
}
We’re talking about batch processing, so I’m not going to go into the form structure or routing here. Let’s look at submitForm() and see what our custom form is doing.
We’re pulling from form_state the value of number_to_execute, which is the number of times we want to execute our batch operation.
Next, we’re initializing our batch array, which we pass into batch_set(). batch_set() is a core function that handles the processing of the batch job. https://api.drupal.org/api/drupal/core%21includes%21form.inc/function/batch_set/8.4.x
Looking at the $batch array, we are setting init_message, which is the message that is displayed while the process initializes.
Next, we define our operations. This is an array of the operations we’re going to run through in our batch job. An operation points to a callback function that will process our batch.
Lastly, we have file, which is the path to the custom module file that contains our operation callback.
batch_set() can handle more parameters, but right now, we’re going to use these three and get a quick batch process running before we go any deeper.
Now that we’ve defined _awesome_batch() as a callback, let’s create it. We say in the file element that there’s an example_batch.awesome_batch.inc file inside our example_batch module. Create that file (modules/custom/example_back/example_batch.awesome.inc):
<?php
use Drupal\Component\Uuid\Php;
/**
* Batch execution.
*
* @param $number
* Number of times to execute.
* @param array $context
* An array of contextual key/value information for rebuild batch process.
*/
function _awesome_batch($number, &$context) {
if ($number && is_numeric($number)) {
$message = \Drupal::translation()->formatPlural(
$number,
'One random string is listed below:', '@count random strings are listed below:'
);
drupal_set_message($message);
// Initiate multistep processing.
if (empty($context['sandbox'])) {
$context['sandbox']['progress'] = 0;
$context['sandbox']['max'] = $number;
}
// Process the next 100 if there are at least 100 left. Otherwise,
// we process the remaining number.
$batch_size = 100;
$max = $context['sandbox']['progress'] + $batch_size;
if ($max > $context['sandbox']['max']) {
$max = $context['sandbox']['max'];
}
// Start where we left off last time.
$start = $context['sandbox']['progress'];
for ($i = $start; $i < $max; $i++) {
$code = randomString();
// We want to display the counter 1-based, not 0-based.
$counter = $i + 1;
drupal_set_message($counter . '. ' . $code);
// Update our progress!
$context['sandbox']['progress']++;
}
// Multistep processing : report progress.
if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
$context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['max'];
}
}
}
/**
* @return string
*/
function randomString() {
$uuid = new Php();
$code = $uuid->generate();
$code = strtoupper($code);
return $code;
}
Our batch operation is pretty simple. We’re going to display a number of random strings (actually UUIDs). How many random strings? However many we got from the form we submitted!
Stepping through _awesome_batch(), let’s start with its arguments.
$number is the number of times we’re going to execute the operation. Again, this is set during the submitForm() in KickOffBatchForm.
$context is an array that the batch process passes in and it has all kinds of interesting parts. The part we’re most concerned about for this particular operation is the sandbox element, but we’ll get to that in a moment.
We’ll check to make sure we’re getting a valid number, then we’ll tell Drupal to print a nice message about how many strings we’re about to generate.
Next, we initialize the parts of the sandbox element that we’re going to use. We’ll use $context[‘sandbox’][‘progress’] so that we can keep track of the iterations we’ve done, and $context[‘sandbox’][‘max’] to show the maximum number of iterations. We’ll use this compute our percent complete and to determine which chunk of operations we’ll do on each call.
Next, we set our $batch_size. We’ll use this to set our batch chunk size to 100 iterations before we finish the function and release control to Drupal’s batch processor so it can update the progress bar and keep our session alive. If we didn’t break up our batch process, we could still time out if it takes longer to process the batch than our allowed php timeout.
Our $max value is the highest number we’ll process in this iteration. If $number is less than the batch size of 100, then it will simply use the $number value that we set into $context[‘sandbox’][‘max’]. If it’s higher than our batch size of 100, then we’ll start by processing a block of 100. Each time we come back, it’ll see if it can do another 100, and if it can’t, it’ll do up to the max value. This means that if our $number is 252, it will first loop 100 times (from 0 to 99), then another 100 times (from 100 to 199), then 52 times (from 200 to 251).
Inside the loop, we can see a call to our randomString() function, which returns a new UUID value. We create a $counter variable so we can display a user-friendly base-1 counter instead of a potentially confusing base-0 counter. Then we use drupal_set_message() to display the counter and the random string.
The last thing our loop does is increment the $context[‘sandbox’][‘progress’] value so that we know where we’ll need to start looping next time.
Once the loop is finished, we need to make sure that Drupal knows whether we’re finished with our operation or not. If we’re not finished, meaning we’ve only done 100 and $number is more than 100, we set $context[‘finished’] to our percent complete. This is a decimal value less than 1. So if $number is 200, at the end of the first iteration, our $context[‘finished’] will be 0.5. When the finished value is 1, Drupal knows that this operation is complete and will either proceed to the next defined operation callback if there is one, or complete the batch job. $context[‘finished’] starts at 1 every time the operation callback is called, so it’s up to us to set it if we’re not finished. If we are finished, we don’t need to do anything. Drupal handles that for us.
Go ahead and run the batch. Make sure you’ve enabled our example_batch module (drush en -y example_batch), go to /example_batch, enter a number in the textbook, and click Submit. You’ll see however many UUIDs that you entered.
Congratulations!
Now, let’s make it a little more complex. We’ll create another form, modules/custom/example_batch/src/Form/MultistepBatchForm.php:
<?php
namespace Drupal\example_batch\Form;
use Drupal\Core\Form\FormBase;
use Drupal\Core\Form\FormStateInterface;
/**
* Class MultistepBatchForm
*/
class MultistepBatchForm extends FormBase {
/**
* {@inheritdoc}
*/
public function getFormId() {
return 'kick_off_batch_form';
}
/**
* {@inheritdoc}
*/
public function buildForm(array $form, FormStateInterface $form_state) {
$form['number_to_execute'] = [
'#type' => 'number',
'#title' => $this->t('Number of times to run'),
'#required' => TRUE,
'#min' => 1,
];
$form['actions'] = ['#type' => 'actions'];
$form['actions']['submit'] = [
'#type' => 'submit',
'#value' => t('Execute'),
];
return $form;
}
/**
* {@inheritdoc}
*/
public function validateForm(array &$form, FormStateInterface $form_state) {
parent::validateForm($form, $form_state);
}
/**
* {@inheritdoc}
*/
public function submitForm(array &$form, FormStateInterface $form_state) {
$number = $form_state->getValue('number_to_execute');
$batch = [
'init_message' => t('Executing a batch...'),
'operations' => [
['\Drupal\example_batch\AnotherBatchOperation::generateSomeStrings', [$number]],
['\Drupal\example_batch\AnotherBatchOperation::reverseThoseStrings', [$number]],
],
'finished' => '\Drupal\example_batch\AnotherBatchOperation::finishUpMyBatch',
];
batch_set($batch);
}
}
And route it (modules/custom/example_batch/example_batch.routing.yml):
example_batch.multistep_form:
path: '/example_multistep'
defaults:
_form: '\Drupal\example_batch\Form\MultistepBatchForm'
_title: 'Multistep Batch Example'
requirements:
_role: 'authenticated'
Clear your cache so Drupal knows about your new route.
Let’s look at submitForm() for our new form. The $batch array is almost the same as before, but we have a couple of small differences. First, we have two operations defined, and they’re in a custom class rather than an include file.
Lastly, we also have defined a finished element. This is a callback that Drupal will call once the batch is complete. We can use this for computing and displaying summary data, cleaning up temp files or private files, or anything else we want to do after the batch operations are complete.
Let’s create that class (modules/custom/example_batch/src/AnotherBatchOperation.php):
<?php
namespace Drupal\example_batch;
use Drupal\Component\Uuid\Php;
class AnotherBatchOperation {
/**
* @param $numberO
* @param $context
*/
public static function generateSomeStrings($number, &$context) {
$message = \Drupal::translation()->formatPlural(
$number,
'One random string is listed below:', '@count random strings are listed below:'
);
drupal_set_message($message);
$context = AnotherBatchOperation::initializeSandbox($number, $context);
$max = AnotherBatchOperation::batchLimit($context);
// Start where we left off last time.
$start = $context['sandbox']['progress'];
for ($i = $start; $i < $max; $i++) {
$str = AnotherBatchOperation::randomString();
$context['results'][] = $str;
// We want to display the counter 1-based, not 0-based.
$counter = $i + 1;
drupal_set_message($counter . '. ' . $str);
// Update our progress!
$context['sandbox']['progress']++;
}
$context = self::contextProgress($context);
}
/**
* @param $number
* @param $context
*/
public static function reverseThoseStrings($number, &$context) {
$message = \Drupal::translation()->formatPlural(
$number,
'The prior random string is listed below, reversed:', 'The prior @count random strings are listed below, reversed:'
);
drupal_set_message($message);
$context = self::initializeSandbox($number, $context);
$max = self::batchLimit($context);
// Start where we left off last time.
$start = $context['sandbox']['progress'];
for ($i = $start; $i < $max; $i++) {
$str = strrev($context['results'][$i]);
// We want to display the counter 1-based, not 0-based.
$counter = $i + 1;
drupal_set_message($counter . '. ' . $str);
// Update our progress!
$context['sandbox']['progress']++;
}
$context = self::contextProgress($context);
}
/**
* @param $context
*
* @return mixed
*/
protected static function contextProgress(&$context) {
if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
$context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['max'];
}
return $context;
}
public function finishUpMyBatch($success, $results, $operations) {
// The 'success' parameter means no fatal PHP errors were detected. All
// other error management should be handled using 'results'.
if ($success) {
$message = \Drupal::translation()
->formatPlural(count($results), 'One string generated and reversed.', '@count strings generated and reversed.');
}
else {
$message = t('Finished with an error.');
}
drupal_set_message($message);
}
/**
* @param $number
* @param $context
*
* @return mixed
*/
protected static function initializeSandbox($number, &$context) {
if (empty($context['sandbox'])) {
$context['sandbox']['progress'] = 0;
$context['sandbox']['max'] = $number;
$context['sandbox']['working_set'] = [];
}
return $context;
}
/**
* @param $context
*/
protected static function batchLimit(&$context) {
// Process the next 100 if there are at least 100 left. Otherwise,
// we process the remaining number.
$batchSize = 100;
$max = $context['sandbox']['progress'] + $batchSize;
if ($max > $context['sandbox']['max']) {
$max = $context['sandbox']['max'];
}
return $max;
}
/**
* @return string
*/
protected static function randomString() {
$uuid = new Php();
$code = $uuid->generate();
$code = strtoupper($code);
return $code;
}
}
Our first operation is \Drupal\example_batch\AnotherBatchOperation::generateSomeStrings(). It’s important to note that the operations callback functions must be static.
The first thing it does is display the number of strings it’s going to generate, just like before.
Next, it sets its $context and $max values from a function on the class, since it hosts multiple operations callbacks and we’re going to do the same processing in both of them.
Now it loops the same as before, except we’re also setting $context[‘results’]. The results element is a special element of $context that Drupal will retain between operations, whether they’re iterations of the same operation or separate operation callbacks. We make an array element in results, then add an element for each random string we generate, so that we can act on those strings in the next operation.
After the loop is done, it calls contextProgress() so the finished element contains the correct percentage of completion.
Once we’ve finished the first operation, Drupal’s batch processing will begin the next operation. We’ve defined this callback as \Drupal\example_batch\AnotherBatchOperation::reverseThoseStrings(), which uses the results generated by the first operation and stored in $context[‘result’] and reverses each one.
This function looks almost identical to generateSomeStrings(). It does all the same $context processing and loops the same way, but inside, we’ll get the element from $context[‘result’] that matches our $i index and reverse it. Then we’ll display that with drupal_set_message(). Otherwise, the function is all of the same stuff we’ve just learned about in our prior operations.
Lastly, the batch processor will call \Drupal\example_batch\AnotherBatchOperation::finishUpMyBatch(). The finish callback gets a different set of arguments: $success, which means no PHP errors were detected; $results, which contains $context[‘result’], and $operations, which contains the operations that haven’t been processed if there is an error and $success is FALSE.
We’re going to check for success, then show a message saying how many strings we generated and reversed.
Now it’s done! We haven’t defined a redirect, so once it’s finished processing, it’ll return to the form we started from and display all of our messages.
I recently had a requirement where I had to generate a large number of nodes and more than a couple hundred would result in a timeout. Changing this process to a batch operation allowed the client to generate as much of this content as they wanted without timing the system out.
You can learn more about batch processing in the documentation on Drupal.org.
Download the code at https://github.com/Breakthrough-Technologies/drupal_example_batch.