Back-End Eng SRE Metrics
Weekly Report
Weekly update for -
8/4/2025
8/11/2025
Total Incidents
0
0
Incidents Prior Week:
0
open incidents!
0
Avg Time to Recover
Avg Time to Recover Last Week:
Avg Time to Ack
Avg Time to Ack Last Week:
This Week's Incident Timeline
Apr 2025
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
May 2025
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Jun 2025
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Jul 2025
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Aug 2025
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Sep 2025
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Oct 2025
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Nov 2025
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Dec 2025
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Month
TodayFit
Count of Incidents by Service
MTTA and MTTR by Team (mins)
This Week's Incidents
Incident Title
Time Created
Service
Latest Status
Assignee
Incident Resolved
Incident Triggered
MTTR
MTTA
URL
TooMany4xxErrors: The partner-file-drop service is serving an unexpected number of 4xx error codes.
7/24/2023, 10:43 AM
Partner File Drop
resolved
Chris Elion
7/24/2023, 10:43 AM
7/24/2023, 10:43 AM
20 secs
10 secs
KafkaDeliverySuccessCountLow: Number of successfully delivered kafka messages is lower than expected.
6/30/2023, 5:58 PM
Telemetry Ingest
resolved
Chris Elion
6/30/2023, 5:58 PM
6/29/2023, 12:04 PM
1 day 5 hrs 54 mins 22 secs
KafkaDeliverySuccessCountLow: Number of successfully delivered kafka messages is lower than expected.
6/30/2023, 5:58 PM
Telemetry Ingest
resolved
Chris Elion
6/30/2023, 5:58 PM
6/29/2023, 12:06 PM
1 day 5 hrs 52 mins 11 secs
RequestRateLow: The request rate for telemetry-ingest is lower than normal.
6/30/2023, 5:58 PM
Telemetry Ingest
resolved
Chris Elion
6/30/2023, 5:58 PM
6/29/2023, 12:06 PM
1 day 5 hrs 52 mins 13 secs
RequestRateLow: The request rate for telemetry-ingest is lower than normal.
6/30/2023, 5:58 PM
Telemetry Ingest
resolved
Chris Elion
6/30/2023, 5:58 PM
6/28/2023, 1:48 PM
2 days 4 hrs 10 mins 14 secs
KafkaDeliverySuccessCountLow: Number of successfully delivered kafka messages is lower than expected.
6/30/2023, 5:58 PM
Telemetry Ingest
resolved
Chris Elion
6/30/2023, 5:58 PM
6/28/2023, 1:48 PM
2 days 4 hrs 9 mins 57 secs
RequestRateLow: The request rate for telemetry-ingest is lower than normal.
6/30/2023, 5:58 PM
Telemetry Ingest
resolved
Chris Elion
6/30/2023, 5:58 PM
6/28/2023, 9:48 AM
2 days 8 hrs 10 mins 30 secs
KafkaDeliverySuccessCountLow: Number of successfully delivered kafka messages is lower than expected.
6/30/2023, 5:58 PM
Telemetry Ingest
resolved
Chris Elion
6/30/2023, 5:58 PM
6/28/2023, 9:48 AM
2 days 8 hrs 10 mins 15 secs
TooMany4xxErrors: The partner-file-drop service is serving an unexpected number of 4xx error codes.
6/30/2023, 5:23 PM
Partner File Drop
resolved
Chris Elion
6/30/2023, 5:23 PM
6/30/2023, 5:23 PM
17 secs
12 secs
KafkaDeliverySuccessCountLow: Number of successfully delivered kafka messages is lower than expected.
6/30/2023, 2:18 PM
Telemetry Ingest
resolved
Chris Elion
6/30/2023, 2:18 PM
6/30/2023, 1:35 PM
42 mins 49 secs
1 min 15 secs
KafkaDeliveryFailureCountHigh: Number of delivery failures for kafka messages is higher than expected.
6/30/2023, 12:21 PM
Telemetry Ingest
resolved
Chris Elion
6/30/2023, 12:21 PM
6/30/2023, 11:29 AM
51 mins 44 secs
23 mins 34 secs
RequestRateLow: The request rate for telemetry-ingest is lower than normal.
6/29/2023, 4:39 PM
Telemetry Ingest
resolved
Chris Elion
6/29/2023, 4:39 PM
6/29/2023, 4:06 PM
32 mins 25 secs
13 mins 36 secs
KafkaDeliverySuccessCountLow: Number of successfully delivered kafka messages is lower than expected.
6/29/2023, 4:39 PM
Telemetry Ingest
resolved
Chris Elion
6/29/2023, 4:39 PM
6/29/2023, 4:04 PM
34 mins 17 secs
15 mins 33 secs
RequestRateLow: The request rate for telemetry-ingest is lower than normal.
6/29/2023, 4:39 PM
Telemetry Ingest
resolved
Chris Elion
6/29/2023, 4:39 PM
6/29/2023, 8:06 AM
8 hrs 32 mins 48 secs
57 secs
KafkaDeliverySuccessCountLow: Number of successfully delivered kafka messages is lower than expected.
6/29/2023, 4:38 PM
Telemetry Ingest
resolved
Chris Elion
6/29/2023, 4:38 PM
6/29/2023, 8:04 AM
8 hrs 34 mins 42 secs
2 mins 49 secs
KafkaDeliverySuccessCountLow: Number of successfully delivered kafka messages is lower than expected.
6/29/2023, 4:38 PM
Telemetry Ingest
resolved
Chris Elion
6/29/2023, 4:38 PM
6/29/2023, 8:06 AM
8 hrs 32 mins 3 secs
46 secs
KafkaDeliverySuccessCountLow: Number of successfully delivered kafka messages is lower than expected.
6/29/2023, 4:38 PM
Telemetry Ingest
resolved
Chris Elion
6/29/2023, 4:38 PM
6/29/2023, 4:06 PM
31 mins 29 secs
13 mins 9 secs
KafkaDeliverySuccessCountLow: Number of successfully delivered kafka messages is lower than expected.
6/28/2023, 5:11 PM
Telemetry Ingest
resolved
Chris Elion
6/28/2023, 5:11 PM
6/28/2023, 1:46 PM
3 hrs 25 mins
46 secs
KafkaDeliverySuccessCountLow: Number of successfully delivered kafka messages is lower than expected.
6/28/2023, 5:11 PM
Telemetry Ingest
resolved
Chris Elion
6/28/2023, 5:11 PM
6/28/2023, 9:46 AM
7 hrs 25 mins 15 secs
36 secs
DBT Failure
Message: Database Error in model miso_dart_lmp_prices_acfg (models/marts/feast/price/miso_dart_lmp_prices_acfg.sql)
001003 (42000): SQL compilation error:
syntax error line 57 at position 4 unexpected 'select'.
syntax error line 59 ...
6/28/2023, 6:50 AM
Infrastructure Operations
resolved
Alejandro Botas
6/28/2023, 6:50 AM
6/22/2023, 12:59 PM
5 days 17 hrs 51 mins 39 secs
DBT Failure
Message: Database Error in model stg_ye_dscloud_yesdata__TS_GEN (models/staging/ye_dscloud/yesdata/stg_ye_dscloud_yesdata__TS_GEN.sql)
001003 (42000): SQL compilation error:
syntax error line 109 at position 0 unexpected 'select'.
c ...
6/28/2023, 6:50 AM
Infrastructure Operations
resolved
Alejandro Botas
6/28/2023, 6:50 AM
6/22/2023, 12:59 PM
5 days 17 hrs 51 mins 39 secs
DBT Failure
Message: [0m21:46:48 Running with dbt=1.4.1
[0m21:46:49 Found 67 models, 432 tests, 0 snapshots, 0 analyses, 727 macros, 0 operations, 0 seed files, 26 sources, 0 exposures, 0 metrics
[0m21:46:49
[0m21:46:53 Concurrency: 4 thread ...
6/28/2023, 6:50 AM
Infrastructure Operations
resolved
Alejandro Botas
6/28/2023, 6:50 AM
6/21/2023, 3:47 PM
6 days 15 hrs 3 mins 43 secs
KafkaDeliveryFailureCountHigh: Number of delivery failures for kafka messages is higher than expected.
6/27/2023, 10:50 PM
Telemetry Ingest
resolved
Chris Elion
6/27/2023, 10:50 PM
6/27/2023, 8:03 PM
2 hrs 47 mins 28 secs
14 mins 41 secs
KafkaDeliverySuccessCountLow: Number of successfully delivered kafka messages is lower than expected.
6/27/2023, 7:02 PM
Telemetry Ingest
resolved
Chris Elion
6/27/2023, 7:02 PM
6/27/2023, 3:41 PM
3 hrs 21 mins 33 secs
1 min 21 secs
RequestRateLow: The request rate for telemetry-ingest is lower than normal.
6/27/2023, 7:02 PM
Telemetry Ingest
resolved
Chris Elion
6/27/2023, 7:02 PM
6/27/2023, 3:43 PM
3 hrs 19 mins 33 secs
12 secs
KafkaDeliverySuccessCountLow: Number of successfully delivered kafka messages is lower than expected.
6/27/2023, 7:02 PM
Telemetry Ingest
resolved
Chris Elion
6/27/2023, 7:02 PM
6/27/2023, 3:43 PM
3 hrs 19 mins 13 secs
31 secs
DBT Failure
Message: Database Error in model miso_dart_lmp_prices_acfg (models/marts/feast/price/miso_dart_lmp_prices_acfg.sql)
001003 (42000): SQL compilation error:
syntax error line 57 at position 4 unexpected 'select'.
syntax error line 59 ...
6/23/2023, 11:22 AM
Infrastructure Operations
resolved
Alex Jacobsen
6/23/2023, 11:22 AM
6/23/2023, 11:12 AM
9 mins 36 secs
DBT Failure
Message: Database Error in model stg_ye_dscloud_yesdata__TS_GEN (models/staging/ye_dscloud/yesdata/stg_ye_dscloud_yesdata__TS_GEN.sql)
001003 (42000): SQL compilation error:
syntax error line 109 at position 0 unexpected 'select'.
c ...
6/23/2023, 11:22 AM
Infrastructure Operations
resolved
Alex Jacobsen
6/23/2023, 11:22 AM
6/23/2023, 11:12 AM
9 mins 32 secs
DBT Failure
Message: Database Error in model miso_dart_lmp_prices_acfg (models/marts/feast/price/miso_dart_lmp_prices_acfg.sql)
001003 (42000): SQL compilation error:
syntax error line 57 at position 4 unexpected 'select'.
syntax error line 59 ...
6/23/2023, 11:21 AM
Infrastructure Operations
resolved
Alex Jacobsen
6/23/2023, 11:21 AM
6/23/2023, 11:05 AM
16 mins 46 secs
DBT Failure
Message: Database Error in model stg_ye_dscloud_yesdata__TS_GEN (models/staging/ye_dscloud/yesdata/stg_ye_dscloud_yesdata__TS_GEN.sql)
001003 (42000): SQL compilation error:
syntax error line 109 at position 0 unexpected 'select'.
c ...
6/23/2023, 11:21 AM
Infrastructure Operations
resolved
Alex Jacobsen
6/23/2023, 11:21 AM
6/23/2023, 11:05 AM
16 mins 44 secs
TooMany4xxErrors: The partner-file-drop service is serving an unexpected number of 4xx error codes.
6/22/2023, 4:30 PM
Partner File Drop
resolved
Chris Elion
6/22/2023, 4:30 PM
6/22/2023, 4:29 PM
1 min 18 secs
1 min 8 secs
DBT Failure
Message: Database Error in model miso_dart_lmp_prices_acfg (models/marts/feast/price/miso_dart_lmp_prices_acfg.sql)
001003 (42000): SQL compilation error:
syntax error line 57 at position 4 unexpected 'select'.
syntax error line 59 ...
6/22/2023, 12:28 PM
Infrastructure Operations
resolved
Alex Jacobsen
6/22/2023, 12:28 PM
6/22/2023, 12:28 PM
29 secs
DBT Failure
Message: Database Error in model stg_ye_dscloud_yesdata__TS_GEN (models/staging/ye_dscloud/yesdata/stg_ye_dscloud_yesdata__TS_GEN.sql)
001003 (42000): SQL compilation error:
syntax error line 109 at position 0 unexpected 'select'.
c ...
6/22/2023, 12:28 PM
Infrastructure Operations
resolved
Alex Jacobsen
6/22/2023, 12:28 PM
6/22/2023, 12:28 PM
24 secs
DBT Failure
Message: Database Error in model miso_dart_lmp_prices_acfg (models/marts/feast/price/miso_dart_lmp_prices_acfg.sql)
001003 (42000): SQL compilation error:
syntax error line 57 at position 4 unexpected 'select'.
syntax error line 59 ...
6/22/2023, 12:23 PM
Infrastructure Operations
resolved
Alex Jacobsen
6/22/2023, 12:23 PM
6/22/2023, 12:16 PM
6 mins 17 secs
DBT Failure
Message: Database Error in model stg_ye_dscloud_yesdata__TS_FLOW (models/staging/ye_dscloud/yesdata/stg_ye_dscloud_yesdata__TS_FLOW.sql)
001003 (42000): SQL compilation error:
syntax error line 109 at position 0 unexpected 'select'.
...
6/21/2023, 5:43 PM
Infrastructure Operations
resolved
Alex Jacobsen
6/21/2023, 5:43 PM
6/21/2023, 5:39 PM
3 mins 39 secs
DBT Failure
Message: Database Error in model stg_ye_dscloud_yesdata__TS_GEN (models/staging/ye_dscloud/yesdata/stg_ye_dscloud_yesdata__TS_GEN.sql)
001003 (42000): SQL compilation error:
syntax error line 109 at position 0 unexpected 'select'.
c ...
6/21/2023, 5:43 PM
Infrastructure Operations
resolved
Alex Jacobsen
6/21/2023, 5:43 PM
6/21/2023, 5:39 PM
3 mins 35 secs
DBT Failure
Message: Database Error in model stg_ye_dscloud_yesdata__TS_FLOW (models/staging/ye_dscloud/yesdata/stg_ye_dscloud_yesdata__TS_FLOW.sql)
001003 (42000): SQL compilation error:
syntax error line 109 at position 0 unexpected 'select'.
...
6/21/2023, 5:39 PM
Infrastructure Operations
resolved
Alex Jacobsen
6/21/2023, 5:39 PM
6/21/2023, 5:29 PM
10 mins 5 secs
DBT Failure
Message: Database Error in model stg_ye_dscloud_yesdata__TS_GEN (models/staging/ye_dscloud/yesdata/stg_ye_dscloud_yesdata__TS_GEN.sql)
001003 (42000): SQL compilation error:
syntax error line 109 at position 0 unexpected 'select'.
c ...
6/21/2023, 5:39 PM
Infrastructure Operations
resolved
Alex Jacobsen
6/21/2023, 5:39 PM
6/21/2023, 5:29 PM
10 mins 1 sec
DBT Failure
Message: Database Error in model stg_ye_dscloud_yesdata__TS_FLOW (models/staging/ye_dscloud/yesdata/stg_ye_dscloud_yesdata__TS_FLOW.sql)
001003 (42000): SQL compilation error:
syntax error line 109 at position 0 unexpected 'select'.
...
6/21/2023, 5:39 PM
Infrastructure Operations
resolved
Alex Jacobsen
6/21/2023, 5:39 PM
6/21/2023, 5:36 PM
2 mins 22 secs
DBT Failure
Message: Database Error in model stg_ye_dscloud_yesdata__TS_GEN (models/staging/ye_dscloud/yesdata/stg_ye_dscloud_yesdata__TS_GEN.sql)
001003 (42000): SQL compilation error:
syntax error line 109 at position 0 unexpected 'select'.
c ...
6/21/2023, 5:39 PM
Infrastructure Operations
resolved
Alex Jacobsen
6/21/2023, 5:39 PM
6/21/2023, 5:37 PM
2 mins 19 secs
DBT Failure
Message: Database Error in model stg_ye_dscloud_yesdata__TS_FLOW (models/staging/ye_dscloud/yesdata/stg_ye_dscloud_yesdata__TS_FLOW.sql)
001003 (42000): SQL compilation error:
syntax error line 109 at position 0 unexpected 'select'.
...
6/21/2023, 5:39 PM
Infrastructure Operations
resolved
Alex Jacobsen
6/21/2023, 5:39 PM
6/21/2023, 5:38 PM
38 secs
DBT Failure
Message: Database Error in model stg_ye_dscloud_yesdata__TS_GEN (models/staging/ye_dscloud/yesdata/stg_ye_dscloud_yesdata__TS_GEN.sql)
001003 (42000): SQL compilation error:
syntax error line 109 at position 0 unexpected 'select'.
c ...
6/21/2023, 5:39 PM
Infrastructure Operations
resolved
Alex Jacobsen
6/21/2023, 5:39 PM
6/21/2023, 5:38 PM
34 secs
TooMany4xxErrors: The partner-file-drop service is serving an unexpected number of 4xx error codes.
6/21/2023, 4:26 PM
Partner File Drop
resolved
Chris Elion
6/21/2023, 4:26 PM
6/21/2023, 4:22 PM
3 mins 20 secs
43 secs
DBT Failure
Message: [0m23:06:39 Running with dbt=1.4.1
[0m23:06:39 Unable to do partial parsing because of a version mismatch
[0m23:06:43 Found 67 models, 419 tests, 0 snapshots, 0 analyses, 727 macros, 0 operations, 0 seed files, 26 sources, ...
6/20/2023, 5:18 PM
Infrastructure Operations
resolved
Alex Jacobsen
6/20/2023, 5:18 PM
6/20/2023, 5:06 PM
11 mins 33 secs
ContainerMemoryUsage: telemetry-ingest - Container Memory usage is too high.
6/20/2023, 3:00 PM
Telemetry Ingest
resolved
Chris Elion
6/20/2023, 3:00 PM
6/20/2023, 2:56 PM
4 mins 18 secs
31 secs
ContainerMemoryUsage: telemetry-ingest - Container Memory usage is too high.
6/20/2023, 3:00 PM
Telemetry Ingest
resolved
Chris Elion
6/20/2023, 3:00 PM
6/20/2023, 2:56 PM
4 mins 15 secs
23 secs
No results from filter
Service
Want to print your doc?
This is not the way.
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.