How to Reduce Procedure Execution Retry Error Notification Noise

Procedures have a step type that allows a user to retry a procedure execution in the case of an error that necessitates re-executing the entire procedure. However, if the email in the Notification Settings for a procedure instance is specified, each procedure execution failure will result in an error notification email, which due to the procedure execution retry step in a procedure will result in a non-actionable notification. Imagine if your procedure executes a 1000 times in a given day and for whatever reason 20% of the executions fail and are retried. This would lead to 200 unnecessary emails. That does not count any execution retries that fail and are retried again.

One could argue to switch off notifications on the procedure instance and use the notify.email() function in a procedure script step. However, that will still end up with unnecessary notifications as the procedure execution is still deemed a failure. So how can this be fixed? Please read on.

Instructions

The following steps should help solve the above problem. Please note, some changes need to be made to your procedure.

Disable notifications on the procedure instance.

If your procedure has notifications enabled, as shown below,

 

disable notifications by removing the email address from the Notification Settings as shown below,


and save the procedure instance.

Now to the procedure changes. The following procedure is one that simulates a step failure and finally executes a Procedure Execution Retry step when the previous step fails, i.e., it re-runs the whole procedure.


We first add a configuration variable to our procedure to set the maximum number of retry attempts for the procedure execution retry. This variable will be used to determine if the procedure is going to retry its execution or if it has run out of execution attempts. The illustrations below show this configuration variable and an example value for the procedure instance.



Pro Tip: When adding a configuration variable to a procedure that has existing instances, you do have to edit the instance.

We now need to add a step, check-retry-attempt-count, between http-request-that-will-fail and the retry-procedure steps.


The check-retry-attempt-count step is a script step, and should be triggered as the onFailure action of the previous step, i.e. the step that fails which results in a procedure execution retry. The Javascript content of the check-retry-attempt-count step is shown below.

check-retry-attempt-count step code
let retryFormulaStep = steps["retry-formula"];

if (retryFormulaStep === undefined ||
    retryFormulaStep === null) {
  throw("retry the formula");
} else {
  if (retryFormulaStep.attempt < config.maxRetryAttempts) {
    throw("retry the formula");
  } else {
    notify.email("name@email.com, "<This is the subject line of the email>", "<This is the error message for the email.>");
  }
}

Let's walk through the above logic.

  1. The script first tries to find the procedure execution retry step by name, in this case retry-formula.
  2. For the very first occurrence of an error in the preceding step (the http-request-that-will-fail step), i.e., when the procedure execution has not been retried as yet, the step value will be undefined or null.
  3. This will result in the check-retry-attempt-count step throwing an error, and the procedure being retried.
  4. If the http-request-that-will-fail step fails once again during the retry attempt, then the check-retry-attempt-count will check the retry attempt number via the attempt attribute of the retry-formula step.
  5. If the attempt number is less than the maximum retry attempts set in the procedure instance configuration, then an error will be thrown by the check-retry-attempt-count step, as in #2 above, which will trigger another retry of the procedure execution.
  6. If the attempt number is equal to the configured maximum retry attempts then a notification email with the error message can be sent to a specific email address (for multiple email addresses, use comma separated email addresses). Note: Although the notification can also be achieved in the last step, i.e., the gracefully-end step, the email recipient and the error message may vary based upon which step resulted in this code path, so the check-retry-attempt-count step can be made specific to the trigger of the error message.

When the number of procedure executions have been exhausted, the check-retry-attempt-count step will end with a success, and can trigger a final script step to gracefully end the procedure execution. In this example, the following code is implemented in the end-gracefully step.

end-gracefully step code
done({ success: false });

In summary, when actionable notification are required from a procedure, the notify.email() function is a better approach to use than enabling the Notifications Settings on a procedure instance.