PyBusMap issueshttps://git.xhost.io/anton/pybusmap/-/issues2017-10-14T17:06:41-04:00https://git.xhost.io/anton/pybusmap/-/issues/12Allow multiple stops in embed URLs2017-10-14T17:06:41-04:00Anton Sarukhanovcode@ant.srAllow multiple stops in embed URLsEmbed URLs should allow users to specify multiple stops. The map should be zoomed and centered such that all stops are visible. Predictions should be shown for all stops (when requested).
UI for generating embed URLs should be updated a...Embed URLs should allow users to specify multiple stops. The map should be zoomed and centered such that all stops are visible. Predictions should be shown for all stops (when requested).
UI for generating embed URLs should be updated accordingly.https://git.xhost.io/anton/pybusmap/-/issues/11Log Nextbus API request roundtrip time2017-10-14T17:06:41-04:00Anton Sarukhanovcode@ant.srLog Nextbus API request roundtrip timeCPU util spikes might be caused by Nexbus API delays? Log request times with ApiCall objects.CPU util spikes might be caused by Nexbus API delays? Log request times with ApiCall objects.https://git.xhost.io/anton/pybusmap/-/issues/10hashchange does not update predictions in combined view2017-10-14T17:06:41-04:00Anton Sarukhanovcode@ant.srhashchange does not update predictions in combined viewWhen an `ec` URL hash is changed, the map view updates but the predictions do not. The predictions half should update as well.When an `ec` URL hash is changed, the map view updates but the predictions do not. The predictions half should update as well.https://git.xhost.io/anton/pybusmap/-/issues/9lat_lon_count increases monotonically2017-10-14T17:06:41-04:00Anton Sarukhanovcode@ant.srlat_lon_count increases monotonicallyWhen `manage.py data_init` is called with a database which already contains stops, those stops' `lat_lon_count` fields will be incremented. That should not happen. If a matching stop is found, either don't update it, or update it more ca...When `manage.py data_init` is called with a database which already contains stops, those stops' `lat_lon_count` fields will be incremented. That should not happen. If a matching stop is found, either don't update it, or update it more carefully.https://git.xhost.io/anton/pybusmap/-/issues/7Regular auto-reload2017-10-14T17:06:41-04:00Anton Sarukhanovcode@ant.srRegular auto-reloadReload the entire page on some timer, perhaps once every 12 hours?
## Reasons
* Bugfixes/updates pushed to non-interactive viewers
* Buggy code can be a server performance drain, need ability to kill it remotely.
## Problems
* P...Reload the entire page on some timer, perhaps once every 12 hours?
## Reasons
* Bugfixes/updates pushed to non-interactive viewers
* Buggy code can be a server performance drain, need ability to kill it remotely.
## Problems
* Push a bad version and non-interactive clients are stuck broken until manual reset
## Improvements
Maybe first do a test request to ensure network connectivity / `200 OK`, then proceed with actual reload.https://git.xhost.io/anton/pybusmap/-/issues/6Max retries exceeded with url (Caused by None)2017-10-14T17:06:41-04:00Anton Sarukhanovcode@ant.srMax retries exceeded with url (Caused by None)To debug, maybe write out a dump of the HTTP request/response?
```
[2016-04-16 12:16:20,094: ERROR/MainProcess] Task celerytasks.update_vehicle_locations[39acb96a-f3dc-428f-b670-327a2d43f61b] raised unexpected: ConnectionError(MaxRet...To debug, maybe write out a dump of the HTTP request/response?
```
[2016-04-16 12:16:20,094: ERROR/MainProcess] Task celerytasks.update_vehicle_locations[39acb96a-f3dc-428f-b670-327a2d43f61b] raised unexpected: ConnectionError(MaxRetryError('None: Max retries exceeded with url: /
service/publicXMLFeed?command=vehicleLocations&t=0&r=kearney&a=rutgers (Caused by None)',),)
Traceback (most recent call last):
File "/home/anton/.virtualenvs/pybusmap/lib/python3.4/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/anton/pybusmap/celerytasks.py", line 27, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/home/anton/.virtualenvs/pybusmap/lib/python3.4/site-packages/flask_celery.py", line 229, in __call__
return task_base.__call__(self, *_args, **_kwargs)
File "/home/anton/.virtualenvs/pybusmap/lib/python3.4/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/home/anton/pybusmap/celerytasks.py", line 74, in update_vehicle_locations
vl_count = len(Nextbus.get_vehicle_locations(agencies, truncate=False))
File "/home/anton/pybusmap/nextbus.py", line 401, in get_vehicle_locations
responses = cls.async_request(requests)
File "/home/anton/pybusmap/nextbus.py", line 108, in async_request
response = f.result()
File "/usr/lib/python3.4/concurrent/futures/_base.py", line 402, in result
return self.__get_result()
File "/usr/lib/python3.4/concurrent/futures/_base.py", line 354, in __get_result
raise self._exception
File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/anton/.virtualenvs/pybusmap/lib/python3.4/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/home/anton/.virtualenvs/pybusmap/lib/python3.4/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/home/anton/.virtualenvs/pybusmap/lib/python3.4/site-packages/requests/adapters.py", line 423, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: None: Max retries exceeded with url: /service/publicXMLFeed?command=vehicleLocations&t=0&r=kearney&a=rutgers (Caused by None)
```https://git.xhost.io/anton/pybusmap/-/issues/5Could not acquire lock: bm-lock-x-vehicle_locations2017-10-14T17:06:41-04:00Anton Sarukhanovcode@ant.srCould not acquire lock: bm-lock-x-vehicle_locationsSeemingly always related to vehicle_locations, never predictions.
```
[2016-04-16 13:04:19,465: WARNING/Worker-2] Got 6 vehicle locations for 1 agencies in 0.98 seconds.
[2016-04-16 13:04:20,922: ERROR/MainProcess] Task celerytasks....Seemingly always related to vehicle_locations, never predictions.
```
[2016-04-16 13:04:19,465: WARNING/Worker-2] Got 6 vehicle locations for 1 agencies in 0.98 seconds.
[2016-04-16 13:04:20,922: ERROR/MainProcess] Task celerytasks.update_vehicle_locations[039f421e-6cee-41c1-a1cf-e499eff5fb81] raised unexpected: LockException('Could not acquire lock: bm-lock-x-vehicle_locations',)
Traceback (most recent call last):
File "/home/anton/.virtualenvs/pybusmap/lib/python3.4/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/anton/pybusmap/celerytasks.py", line 27, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/home/anton/.virtualenvs/pybusmap/lib/python3.4/site-packages/flask_celery.py", line 229, in __call__
return task_base.__call__(self, *_args, **_kwargs)
File "/home/anton/.virtualenvs/pybusmap/lib/python3.4/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/home/anton/pybusmap/celerytasks.py", line 74, in update_vehicle_locations
vl_count = len(Nextbus.get_vehicle_locations(agencies, truncate=False))
File "/home/anton/pybusmap/nextbus.py", line 375, in get_vehicle_locations
with Lock("agencies", shared=True), Lock("routes", shared=True), Lock("vehicle_locations"):
File "/home/anton/pybusmap/lock.py", line 66, in __enter__
raise(LockException("Could not acquire lock: {0}".format(cls.exclusive_key)))
lock.LockException: Could not acquire lock: bm-lock-x-vehicle_locations
[2016-04-16 13:04:22,012: WARNING/Worker-5] Got 240 predictions for 1 agencies in 1.09 sec.
```https://git.xhost.io/anton/pybusmap/-/issues/4Celery - lock.py - NoneType does not support the buffer interface2017-10-14T17:06:41-04:00Anton Sarukhanovcode@ant.srCelery - lock.py - NoneType does not support the buffer interfaceThis happens occasionally, during otherwise-normal operation.
```
[2016-04-16 13:04:22,012: WARNING/Worker-5] Got 240 predictions for 1 agencies in 1.09 sec.
[2016-04-16 13:04:23,938: ERROR/MainProcess] Task celerytasks.update_vehic...This happens occasionally, during otherwise-normal operation.
```
[2016-04-16 13:04:22,012: WARNING/Worker-5] Got 240 predictions for 1 agencies in 1.09 sec.
[2016-04-16 13:04:23,938: ERROR/MainProcess] Task celerytasks.update_vehicle_locations[6e1de22f-146c-4f0e-96fa-06c67a06c4ff] raised unexpected: TypeError("'NoneType' does not support the buffer interface",)
Traceback (most recent call last):
File "/home/anton/.virtualenvs/pybusmap/lib/python3.4/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/anton/pybusmap/celerytasks.py", line 27, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/home/anton/.virtualenvs/pybusmap/lib/python3.4/site-packages/flask_celery.py", line 229, in __call__
return task_base.__call__(self, *_args, **_kwargs)
File "/home/anton/.virtualenvs/pybusmap/lib/python3.4/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/home/anton/pybusmap/celerytasks.py", line 74, in update_vehicle_locations
vl_count = len(Nextbus.get_vehicle_locations(agencies, truncate=False))
File "/home/anton/pybusmap/nextbus.py", line 375, in get_vehicle_locations
with Lock("agencies", shared=True), Lock("routes", shared=True), Lock("vehicle_locations"):
File "/home/anton/pybusmap/lock.py", line 56, in __enter__
(existing_expires, existing_pid) = pickle.loads(oldlock)
TypeError: 'NoneType' does not support the buffer interface
```https://git.xhost.io/anton/pybusmap/-/issues/3Fetching predictions hangs sometimes2017-10-14T17:06:41-04:00Anton Sarukhanovcode@ant.srFetching predictions hangs sometimesFetching predictions and vehicle locations normally takes ~1 second or less. Sometimes, it can take 5, 10, even 20 seconds to complete.
As there are database locks involved, this delays all other scheduled tasks too, and predictions b...Fetching predictions and vehicle locations normally takes ~1 second or less. Sometimes, it can take 5, 10, even 20 seconds to complete.
As there are database locks involved, this delays all other scheduled tasks too, and predictions become needlessly stale.
## Possible ways to fix
* Reduce/avoid database lock time
* What's causing the delay? API lag? Deadlocks?
Deadlocks should throw an exception, so probably not.
## Example
```
[21:18:21,174: w3] Got 489 predictions for 1 agencies in 1.10 sec.
[21:18:25,979: w2] Got 30 vehicle locations for 1 agencies in 4.80 seconds. !!
[21:18:26,747: w4] Got 30 vehicle locations for 1 agencies in 0.76 seconds.
[21:18:37,518: w5] Got 30 vehicle locations for 1 agencies in 17.45 seconds. !!!
[21:18:38,272: w5] Got 30 vehicle locations for 1 agencies in 0.75 seconds.
[21:18:38,861: w3] Got 490 predictions for 1 agencies in 1.34 sec.
[21:18:39,215: w2] Got 30 vehicle locations for 1 agencies in 0.94 seconds.
[21:18:40,216: w5] Got 30 vehicle locations for 1 agencies in 1.94 seconds.
[21:18:42,055: w3] Got 490 predictions for 1 agencies in 1.84 sec.
[21:18:42,079: w5] Got 30 vehicle locations for 1 agencies in 1.86 seconds.
[21:18:43,510: w5] Got 30 vehicle locations for 1 agencies in 1.43 seconds.
[21:18:44,416: w5] Got 30 vehicle locations for 1 agencies in 0.90 seconds.
[21:18:46,080: w3] Got 489 predictions for 1 agencies in 1.57 sec.
[21:18:58,937: w2] Got 30 vehicle locations for 1 agencies in 16.88 seconds. !!!
[21:18:59,710: w2] Got 30 vehicle locations for 1 agencies in 0.77 seconds.
[21:19:00,929: w5] Got 30 vehicle locations for 1 agencies in 1.22 seconds.
[21:19:01,975: w4] Got 30 vehicle locations for 1 agencies in 23.11 seconds. !!!
[21:19:02,265: w3] Got 489 predictions for 1 agencies in 1.33 sec.
[21:19:03,290: w4] Got 30 vehicle locations for 1 agencies in 1.31 seconds.
[21:19:04,764: w4] Got 30 vehicle locations for 1 agencies in 1.47 seconds.
[21:19:06,148: w3] Got 489 predictions for 1 agencies in 1.38 sec.
[21:19:06,330: w4] Got 30 vehicle locations for 1 agencies in 1.56 seconds.
[21:19:10,568: w4] Got 30 vehicle locations for 1 agencies in 3.10 seconds.
[21:19:12,593: w2] Got 30 vehicle locations for 1 agencies in 10.61 seconds. !!!
[21:19:13,299: w2] Got 30 vehicle locations for 1 agencies in 0.70 seconds.
[21:19:13,489: w3] Got 489 predictions for 1 agencies in 0.89 sec.
[21:19:14,530: w5] Got 30 vehicle locations for 1 agencies in 11.24 seconds. !!!
[21:19:15,308: w5] Got 30 vehicle locations for 1 agencies in 0.78 seconds.
[21:19:16,229: w2] Got 30 vehicle locations for 1 agencies in 2.75 seconds.
```
https://git.xhost.io/anton/pybusmap/-/issues/2Failed stops/routes refresh breaks everything?2017-10-14T17:06:41-04:00Anton Sarukhanovcode@ant.srFailed stops/routes refresh breaks everything?If a scheduled (celery) refresh of Stop/Route data fails, the database can be left with no data. Happened once in development; haven't witnessed it yet in production.
Keeping an eye out and using this issue to track observations...If a scheduled (celery) refresh of Stop/Route data fails, the database can be left with no data. Happened once in development; haven't witnessed it yet in production.
Keeping an eye out and using this issue to track observations...