Resolved -
2/5 Payouts processed as expected with increased resources.
Feb 6, 01:44 UTC
Update -
We have increased the resources available to the daily payout process for tonight's payout window. The payout process will be observed tonight to ensure a smooth operation.
The JustiFi team has identified a potential solution to reduce the resources consumed by this tasks and will be working to get that deployed during operating hours tomorrow.
Feb 5, 22:59 UTC
Monitoring -
A resource limitation has been identified in our automated payout system. On Feb 4th at 1:30UTC, our process began working on a queue of pending payouts and started sending the requests to our payout provider API around 1:50UTC we started experience some slowness due to a resource constraint on the task. This slowdown continued until 4:00UTC where the task fully paused. At 11AM UTC, the task was recycled and the payout queue began processing again. At 11:03UTC all payouts completed processing.
This delay meant that the effective entry date for the payouts which processed after the slowdown are entered effectively as of the next business day, rather than the current day.
The JustiFi engineering team is increasing the resources available to the payout system and also working on improvements for the resource usage of this critical task.
Feb 5, 20:48 UTC