Decision Cloud Status

Delayed Reports

Update - We have increased the number of processes working on the backlog of performance data and are seeing the queue catching up once more, we anticipate the reports being up to date within 30 minutes.
Apr 25, 2024 - 07:32 PDT

Monitoring - The queue blocker has been cleared and reports from daily files and this morning are filling in.
Apr 25, 2024 - 05:41 PDT

Investigating - Investigating backlog in the reporting queue
Apr 25, 2024 - 05:19 PDT

About This Site

This is our Decision Cloud status page, where you can get updates on how our systems are doing. If there are interruptions to service, we will post a note here.

As always, if you are experiencing any issues with Decision Cloud, don't hesitate to get in touch with us at support@insight.tm and we'll get back to you as soon as we can.

Uptime over the past 90 days. View historical uptime.

Decision Cloud - Web Site Operational

Decision Cloud - API (Lead Processing) Operational

Decision Cloud - Reports Degraded Performance

BigML Decisions Operational

File Builder Operational

90 days ago

100.0 % uptime

Today

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

System Metrics Month Week Day

Reporting Delay

Fetching

Decision Cloud - Response Time

Fetching

Past Incidents

Apr 25, 2024

Unresolved incident: Delayed Reports.

Apr 24, 2024

No incidents reported.

Apr 23, 2024

Reboot of Database Complete

Resolved - This is an informational notice. No issues at this time.

We conducted a brief reboot of the database to address some immediately required parameter updates. This lasted roughly one minute and is complete. with 4 minute of API Ingestion downtime.

This should represent the last need to interrupt operations related to the upgrade.
Apr 23, 17:38 PDT

File Builder Disabled

Resolved - File builder services have been restored.

Auto Pull files were triggered for today and will complete later today.
Apr 23, 13:23 PDT

Identified - File Builder has been disabled at this time as we are prioritizing API ingestion.
Apr 22, 08:18 PDT

API TruScore Errors

Resolved - This incident has been resolved.
Apr 23, 09:17 PDT

Identified - We have detected and identified an error in TruScore processing.

A patch is being deployed and will be live within 8 minutes.
Apr 23, 09:09 PDT

API Processing Impacted

Resolved - This incident has been resolved.
Apr 23, 08:12 PDT

Update - Ruleset and Price Reject Updates:
A patch has been applied fixing the issue with most rulesets, specifically those using Service Result in rules. We are keeping an eye out for other ruleset issues and have a patching methodology in place.

A fix was supposed to complete this morning before traffic. Unfortunately, the unrelated spike in processing time caused safeguards to trigger canceling the process.

We apologize that it did not complete for the early morning traffic.

All API services online and processing.
Apr 23, 08:12 PDT

Update - Server restart was conducted and metrics are returning to normal levels.

Leads Flowing normally at this time.
Apr 23, 06:21 PDT

Update - We are investigating a spike in processing times and high error rates.
Apr 23, 06:10 PDT

Update - Roughly 50% of the ruleset conditions have been fixed.

We have processes running through the night to continue fixing assorted ruleset conditions and will provide further updates in the AM.
Apr 22, 22:40 PDT

Identified - We have identified the cause of the issues with rulesets, specifically price reject, and are working toward a solution.
Apr 22, 04:40 PDT

Investigating - We are investigating impacts to API processing.

We have confirmed rulesets are impacted at this time and are looking into the situation further.
Apr 22, 03:26 PDT

Reporting Processing Delayed

Resolved - Reporting is 97% up to date for today with the some leads still filling in from the previous hour.

Reports processed daily files last night.

We will rerun lead information from Saturday this evening to make sure all information is for Saturday -> Monday is up to date.
Apr 23, 07:51 PDT

Monitoring - Reports are catching 90% caught up now with some leads from the 8 AM hour to fill in within 30 minutes.
Apr 23, 07:06 PDT

Investigating - Investigating report lag from the last hour.
Apr 23, 06:50 PDT

Monitoring - Performance improvements on the upgraded database have been implemented and reporting information is being processed.

Nightly files will process tonight as usual.

We appreciate the understanding during this upgrade process.
Apr 22, 22:39 PDT

Update - We are working on restoring reports for the day.

Nightly files from Sunday will not be processed until the evening.
Apr 22, 04:40 PDT

Identified - We have identified a delay with nightly file and lead report processing.

We are working to resolve the backlog.

All API and APP functionality is live at this time.
Apr 21, 23:21 PDT

Apr 22, 2024

Apr 21, 2024

Database Upgrade Continued

Completed - The scheduled maintenance has been completed.
Apr 21, 23:00 PDT

Verifying - We are conducting further validations on the database upgrade at this time. Services except API Ingestion are live but we recommend refraining from use until validations are complete.
Apr 21, 19:38 PDT

Update - We have conducted initial swap of our database and are conducting validations. We recommend avoiding application use at this time.
Apr 21, 18:09 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 21, 15:00 PDT

Scheduled - As it is a priority to provide normal lead operations through core hours on Sunday, the decision was made to defer completion of the database upgrade.

The upgrade process has been refined and we will be conducting a maintenance window with the goal of completing this major upgrade.

Estimated Downtime:
* API Ingestion: 6 hours
* APP: 2 hours
* File Builder: 2 hours
* Reports: 1 hours
* API Redirects: 1 hour
Apr 21, 03:58 PDT

API Ingestion Still Offline

Resolved - API Ingestion has been live for the 5 oclock CST hour. All other services are also operational.

Nightly files will be delayed for Sunday as they began processing later than usual.
Apr 21, 03:52 PDT

Identified - Previous maintenance ticket closed preemptively.

We are still working on restoring API Ingestion. All other services are live once more.
Apr 21, 03:05 PDT

Database Upgrade

Completed - The scheduled maintenance has been completed.
Apr 21, 03:00 PDT

Verifying - Database upgrades have been deferred at this time and services are being restored. Nightly files are being processed as usual. API ingestion will be brought online within the hour.
Apr 21, 02:16 PDT

Update - Maintenance tasks related to database upgrade are taking longer than scheduled.

All services except API Ingestion are still live at this time.

An update on timeline will be provided as available.
Apr 20, 20:17 PDT

Update - All items are proceeding as planned.
Apr 20, 17:14 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 20, 15:00 PDT

Scheduled - Database upgrades will be conducted during this maintenance window. Mandated upgrades to improve performance, increase durability, and provide optimal ongoing service will be applied.

Estimated Maximum Downtime:
* API Ingestion: 6 hours
* APP: 2 hours
* File Builder: 2 hours
* Reports: 1 hours
* API Redirects: 1 hour
Apr 19, 10:50 PDT

Apr 20, 2024

Apr 19, 2024

No incidents reported.

Apr 18, 2024

No incidents reported.

Apr 17, 2024

Database Upgrade Validations

Completed - Maintenance has been completed and main API ingestion confirmed functional.
Apr 17, 23:40 PDT

Update - Main API ingestion will remain disabled for approximately 30 additional minutes. All other services unaffected.
Apr 17, 22:58 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 17, 22:00 PDT

Scheduled - Database validations will be performed during this window. Lead ingestion may be impacted for up to 30 minutes. All other services will experience less than 15 minutes of downtime.
Apr 17, 09:19 PDT

Apr 16, 2024

No incidents reported.

Apr 15, 2024

No incidents reported.

Apr 14, 2024

No incidents reported.

Apr 13, 2024

No incidents reported.

Apr 12, 2024

No incidents reported.

Apr 11, 2024

Database Maintenance

Completed - Maintenance completed ahead of schedule.
Apr 11, 21:24 PDT

Scheduled - We will be undergoing database maintenance at this time. Users may experience intermittent slow lead processing or lack of leads within this window.
Apr 3, 12:43 PDT

About This Site

Related