Fetching predictions hangs sometimes
Fetching predictions and vehicle locations normally takes New Feature second or less. Sometimes, it can take 5, 10, even 20 seconds to complete.
As there are database locks involved, this delays all other scheduled tasks too, and predictions become needlessly stale.
Possible ways to fix
- Reduce/avoid database lock time
- What's causing the delay? API lag? Deadlocks?
Deadlocks should throw an exception, so probably not.
Example
[21:18:21,174: w3] Got 489 predictions for 1 agencies in 1.10 sec.
[21:18:25,979: w2] Got 30 vehicle locations for 1 agencies in 4.80 seconds. !!
[21:18:26,747: w4] Got 30 vehicle locations for 1 agencies in 0.76 seconds.
[21:18:37,518: w5] Got 30 vehicle locations for 1 agencies in 17.45 seconds. !!!
[21:18:38,272: w5] Got 30 vehicle locations for 1 agencies in 0.75 seconds.
[21:18:38,861: w3] Got 490 predictions for 1 agencies in 1.34 sec.
[21:18:39,215: w2] Got 30 vehicle locations for 1 agencies in 0.94 seconds.
[21:18:40,216: w5] Got 30 vehicle locations for 1 agencies in 1.94 seconds.
[21:18:42,055: w3] Got 490 predictions for 1 agencies in 1.84 sec.
[21:18:42,079: w5] Got 30 vehicle locations for 1 agencies in 1.86 seconds.
[21:18:43,510: w5] Got 30 vehicle locations for 1 agencies in 1.43 seconds.
[21:18:44,416: w5] Got 30 vehicle locations for 1 agencies in 0.90 seconds.
[21:18:46,080: w3] Got 489 predictions for 1 agencies in 1.57 sec.
[21:18:58,937: w2] Got 30 vehicle locations for 1 agencies in 16.88 seconds. !!!
[21:18:59,710: w2] Got 30 vehicle locations for 1 agencies in 0.77 seconds.
[21:19:00,929: w5] Got 30 vehicle locations for 1 agencies in 1.22 seconds.
[21:19:01,975: w4] Got 30 vehicle locations for 1 agencies in 23.11 seconds. !!!
[21:19:02,265: w3] Got 489 predictions for 1 agencies in 1.33 sec.
[21:19:03,290: w4] Got 30 vehicle locations for 1 agencies in 1.31 seconds.
[21:19:04,764: w4] Got 30 vehicle locations for 1 agencies in 1.47 seconds.
[21:19:06,148: w3] Got 489 predictions for 1 agencies in 1.38 sec.
[21:19:06,330: w4] Got 30 vehicle locations for 1 agencies in 1.56 seconds.
[21:19:10,568: w4] Got 30 vehicle locations for 1 agencies in 3.10 seconds.
[21:19:12,593: w2] Got 30 vehicle locations for 1 agencies in 10.61 seconds. !!!
[21:19:13,299: w2] Got 30 vehicle locations for 1 agencies in 0.70 seconds.
[21:19:13,489: w3] Got 489 predictions for 1 agencies in 0.89 sec.
[21:19:14,530: w5] Got 30 vehicle locations for 1 agencies in 11.24 seconds. !!!
[21:19:15,308: w5] Got 30 vehicle locations for 1 agencies in 0.78 seconds.
[21:19:16,229: w2] Got 30 vehicle locations for 1 agencies in 2.75 seconds.