Incident Summary
Custom Reports in IRIS without a date range (i.e. very large reports because they’d include all dates) could not be downloaded. Attempting to do so resulted in an IRIS error page and failed download.
Leadup
There was no code change or development initiative that caused the error. There was, however, a period of maintenance that was conducted on servers used by IRIS in the 1-2 weeks prior to receiving the first Support ticket about this issue. We suspect that the accumulation of large amounts of data over time combined with an asynchronous change to a process between servers and the application resulted in this error.
Fault
The function of downloading custom reports from IRIS did not work as originally intended. This is was not the specific fault of any party, but rather the result of a large amount of historical data and server maintenance.
Impact
This issue lasted for 11 days, from October 22nd to November 2nd, and prevented stakeholders from downloading large custom reports. Every client organization in IRIS was affected, but the error only affected users trying to access a very large custom report.
Detection
This issue was raised initially through an affiliate ticket submitted to Support. Due to the isolated nature of this issue and how few users were affected, it may have been difficult to detect the error prior to an affiliate submitting a ticket.
Response
The Support, Command Center and PUMA teams responded to this issue. Once the issue was detected, there were no delays in responding to the incident and the process worked well.
Recovery
Functionality was restored by a fix implemented in IRIS code, which was deployed with release 5.20 on 10/2/20. To resolve this issue, Support informed PUMA and PUMA informed the IRIS developers at iTrans. iTrans developers quickly identified the problem and implemented a fix.
Timeline
October 21st, 7:24am EST – received the first support ticket notifying us of the issue. The Support team begins troubleshooting with the reporter.
October 22nd, 5:18am EST – received an additional support ticket. Support continues troubleshooting and reaches out to the PUMA team.
October 22nd, 12:37pm EST – PUMA sends iTrans team (who is at this time tasked on WMC) an email notifying them of the issue, however, the workday in Belarus, where iTrans is located, is over at this time.
October 23rd, 4:19pm EST – PUMA sends iTrans team a follow-up email due to a delayed response.
October 23rd, 4:31pm to 5pm EST – iTrans responds (after-hours) and clarifies priority with the Product Owner in a series of emails.
October 28th, 10:29am EST – iTrans notifies the PUMA team of the fix for the issue being deployed to the Staging environment. Several emails are changed and further discussion of the deploy to Production with an upcoming release (5.20) is determined.
November 2nd, 12:00am to 1:00am EST – The fix is deployed to Production with the IRIS v5.20 release.
November 2nd, 8:30am EST – PUMA and Support verify that the fix is functioning as designed. Errors and failed downloads when attempting to download custom reports are resolved.
Root Cause
Custom Reports in IRIS produced an error on attempted downloaded because the open date-range of a search pulled in a much larger amount of data than was ever anticipated by development.
Recurrence
This issue has not occurred before.
Lessons Learned
We learned that server maintenance with older systems can result in unanticipated OutOfMemory issues when pulling data with large date ranges in IRIS.
Communication between Support and PUMA, and initial troubleshooting conducted by Support, both went well. The resolution of the issue also went well, once priority and the deploy were determined. However, the response time of developers could be improved.
We got “lucky” in the sense that this fix could not be tested on any environment other than Production, and successfully resolved the issue on the first try.
Corrective actions
Since this issue has not ever occurred before, we do not anticipate it happening again, however, we will conduct additional testing of this feature (Custom Reports) in the future, when there is a period of server maintenance. This will be completed by the entire PUMA team and IRIS developers.