News from the dashboard project developers http://forum.vces.net/index.php Thu, 09 Sep 2010 00:55:56 +0100 15 http://forum.vces.net/index.php?showtopic=5467 http://dashboard.bastage.net

However, due to an ongoing issue with concurrent database queries failing due to transaction deadlocks, development of new features has been suspended pending the rewrite of the core engine.

Signatures and statistics collection from MTA will continue working. However, the live stats page may randomly disconnect you with a server busy message. Please do not raise this is a bug; it's a known issue.

Update 2010-07-17 02:15BST:
The new logging engine is running and collecting statistics and live status from MTA succesfully. The system can now tolerate a downtime period of 30 seconds (whilst the logging engine restarts) in the event of a failure of configuration change, and no data will be lost during these 30 seconds.

Addtionally, the new caching engine is up and running for data on the home page only. The home page of dashboard no longer communicates with a SQL database - it pulls data from an in memory cache instead.

More updates and improvements will be made this weekend.

Update 2010-07-19 01:16BST:
The logging engine has been further streamlined to support the concurrent decoding and logging of messages from MTA. The process is now split into four parts; the transmission of the message from MTA to the dashboard web application via HTTP, the transmission from the web application into the processing engine via WCF, the decoding of the message by a collection of IMtaMessageParser objects and then the updating of relevant caches and database by a collection of IDataOperationHandler objects.

It's all very long winded but effectively means we can send thousands of messages per minute into the engine and it'll process them in it's own time - there's no dependence on SQL Server, processing of messages won't affect the ability for dashboard to show data on the site, so everyone's a winner.

Tomorrow evening will be the transition of more MtaMessages from the old system into the new engine, and improvements to the caching engine.

Estimates are that we'll be able to handle around 10 million requests per hour on the new system.

Update 2010-07-23 00:12BST:
Migration of all the data collection functionality to the new pipeline has been completed. We're now processing all messages from MTA in a multi-threaded, asynchronous fashion meaning we tell MTA that message submission is completed once the message has entered the pipeline. We can then take as long as necessary to process the message and MTA is non the wiser.

The caching engine is also running and stable - all new messages are summarised and stored in cache, and also commited to database. When the engine starts, it retrieves raw stats from the database and performs calculations on these to populate the cache. The result is that after approximately 30 seconds after the engine is running, any data that dashboard needs to access is stored in RAM. This means we don't have to worry about transaction locks on the database.

More testing and implementation of some MTA "talkback" will take place this weekend.

Update 2010-07-24 03:02BST:
More late night coding! MTA now submits messages directly into the processing pipeline via WCF - it no longer needs to be able to talk to dashboard.bastage.net. The upshot of this is that the web interface now does *nothing* except show statistics to users; no processing, no relaying and no caching takes place in the UI layer any more.

I also implemented multiple server support this evening to permit RvB to log statistics as and when the time is right.]]> Wed, 14 Jul 2010 01:12:03 +0100 http://forum.vces.net/index.php?showtopic=5467