Changes to Data Warehouse Upload
We are extremely grateful to all the groups and individuals who have uploaded data to the warehouse and the totals on the upload ranking page are quite amazing.
Since launch we have been monitoring the service which receives, processes and stores that data and now are about to make some changes to it to improve it performance and reliablity.
The first change is to introduce an acceptance time window for packets uploaded from the Dashboard. This restricts packets to those whose sequence numbers are within +/- 48 hours of the latest sequence number stored in the warehouse. We are doing this for two reasons:
- We have received erroneous packets whose sequence numbers are significantly different to the latest value.
- To reduce the chance of double scoring of points when we release stored data back to the user community
The second change, which will not affect normal operation but will improve scoring:
- We currently process packets immediaely they arrive at the server. Under load, this can cause us to lose a few packet scores because of contention in the database. To alleviate this problem, we are going to change the processing slightly to process packets through a buffer. This buffer will be processed every 5 seconds, so there will be a slight delay
before packets are acknowledged on the ranking.
– Dave, G4DPZ