Latest News from StockWave Software Ltd

Latest News

29 April 2016

Windows 10 has continued to prove problematic for StockWave - in light of these "issues" we have decided to turn what is a large, monolithic application into a dozen or so smaller applications, which can run both independently or in combination with each other. The longer term goal of this process is to make the application truly cross-platform. An initial foray into mobile development for Android will be available soon - it will be called StocksBrowser and it - "does what it says on the tin" - it will be a small, but useful stock market application, and will become the basis for much more advanced features, with heavy compute intensive algorithms, e.g. monte carlo simulation, global optimisation, being moved to the cloud. Many apologies to anyone who has visited our site and expected to be able to download trial software - we have had problems and now we are moving in a new direction, but we will be back!

28 August 2014

Some buggers hacked our installers to distribute malware using a cheap bit of php and access to the public ftp account which previously allowed users to download price and news data - so, now NO downloadable trialware anymore, and no public data access (- Google has listed us as a "malware" site so I doubt we'd be getting any traffic anyway); if you want a trial, email us and we'll send you a disk. This is what the world has come to ... oy vey ... its the Tribulation I tells ya ... the end times, the book of daniel...

12 April 2014

The items referred to in the future developments page are more or less, finished, apart from the experimental neural network architectures which remain experimental, not for the retail user; the current trial software will not be upgraded with the new features as we need to incentivise our paying customers - not everything can be free, but hey, the free stuff ain't bad.

Some discoveries we have made - the advanced analyzers continue to be both beguiling and fickle; the first thing you have to do is to find some parameter set which at least has the potential to model the data - you do this using the modelbuilder form, and yes, there will be quite a bit of to-ing and fro-ing and adjustment, it's just the way it is; once you've found a viable parameter set you can go ahead and create your probability heatmaps for the future price moves - these can give fairly accurate short term movements but will fail at times for reasons we understand, and at very long times they can no longer be considered reliable - there the best thing to do is to revert to the basic random walk analyzer. The cleverest strategy to do is to create multiple resolution analyzers which are continually re-calculated over time - and this is something that can be done in Python with some custom programming.

The "reasons we understand" referred to above are our old friends, "trendbreaker" news events - chances are, if the price is moving away from where it should be, then "something" has happened, which you will probably see on the event stream below the price chart if you look. It would be nice if this was not the case, but really, it just validates what we have been saying all along - our original prognostications are entirely correct - and thus to augment our advanced analyzers we need our news event modelling and our data fusion - you cannot get away with only using a subset of our tools.

There is a new dataminer / market scanner which uses free fundamentals from Yahoo and Quandl; the use of this is, I suppose, to act as a kind of preprocessing filter as to what kinds of stocks might be interesting to trade on; the clustering algorithms produce an indication of to what extent stocks are "like" or unlike each other, and this may induce your curiosity one way or the other - I suppose the thing is to try and find something that everyone else has missed. Anyway, it is there, done, available for you - personally I think "fundamentals" are a load of alice-in-wonderland manipulations, exponentiated fictions, but to those people out there who just can't see past their P/E and ROA, there you have it.

Analyzing data can produce visualisations of exquisite beauty- kohonen mappings look fantastic as do the "DNA" maps from bayesian correlators, and when you see a heatmap which shows a definite trend or trajectory, it is very exciting.

The trade creator has been revamped visually to be less confusing, but the one big new thing we're actually quite proud of is the new automatic trade searcher; honestly people, this is worth double the price of the ticket on its own ! - if you don't think this is cool then we'll just ... kill ourselves ... or else you can sod-off back to playing minecraft or angry birds .. anyway ... you can now search exhaustively or for specific known strategies, it can download option chains automatically from yahoo, and can accept CSV format to be copy-pasted - so no longer do you have to fill in tedious rows of numbers. The best new thing though is the optimising search - this uses a genetic algorithm to search potentially huge search spaces, for example 6 trade combinations for an options chain of 20 strikes, using calls and puts, is about 300 million possibilities - you cannot simply enumerate through them one by one ... the details of genetic algorithms can be looked up elsewhere (just google it) but this kind of application is exactly the kind of thing they are meant for, and they work well ... they can find combination trades which are just absolutely "crazy" looking - the other day I found a 99.09% profit probability, with a nice expected return for a 7 trade combination with a profile which was not of any known type ... - the point is this, if you can effectively search a huge search space, you will be looking more or less on your own, and so you can find things no one else has found - single and two-leg option trades are bound to pretty efficiently priced - but because of the increasing exponential complexity of the possible trading space, an undiscovered country of potential profit opens up; this is important because it gives back hope to the retail investor, i.e. in the past few years it has seemed to be that the market prices are being created by rigging at both ultra-short and macro-long range - you have the HFT boys front running everyone else with their collocation, juiced hardware and microwave links, and then you have the Fed with its policy of "quantitative easing" and also of giving out preferential information to the "bigboys" - but since these two, dominant, information streams are not available to the private retail investor, it mean that in this game he is is little more than a "mug punter", or a lamb to the slaughter.

Interactive Brokers datafeed is now also supported.

The python programming interface works well - so now you have full ability to create whatever system you want.

18 March 2013

We are adding support for Interactive Brokers datafeed and brokerage inside StockWave itself; we are looking into some others as well, subject to user request.

Some thoughts on the dirty business of marketing, a current fad and terminology.

22 December 2012
Still here?! ... damn those unreliable Mayans

And life carries on, so enjoy the new website and download the trial version of StockWave; when you want to buy send us an email.

21 December 2012
Largely unordered reflections and musings on a software project called StockWave

Software development is time-consuming, boring, error-prone and with a good chance of outright failure; it is wise to do as little of it as possible, or get someone else to do it, who you can blame if it all goes wrong while having already paid yourself handsomely out of the contract money ... but this is not a tutorial on business sleaze, so we'll leave that alone.
The only things predictably do-able are things very similar to things already done, which is a waste of time, ergo anything worth doing is highly unpredictable; this is the essential problem with software project management, one which no methodology or framework can fix.
There is a massive amount of work to progress along the line of - a good idea, a commercial idea, an implementation, a working prototype, an alpha product, a beta product, a polished high quality consumer product, one which sells, one which makes money. The costs involved in moving along this progression will be wildly underestimated. Having a good idea to begin with is beyond most people, and even if you do have one you are unlikely to have it to yourself for very long.
Advanced analytical techniques which require heavy computations are antithetical to good user experiences in a software application - this presented difficulties to myself as the whole point was to give "advanced algorithms" to the masses in the StockWave project - "every man a quant", "every man his own hedge fund", "everyone making sophisticated plays" ...
Users should never be annoyed, confused, made to wait more than 0.5 seconds - software is there to provide some service to the user to make his life better, not to induce psychosis; the blue screen, the error message dialog box, the spinning hourglass, the round spinny thing, the "xxxx is not responding" message ... they must be removed from human experience.
Multiple processors/cores and threads appear to be the solution, or at least an alleviation of the heavy processing versus user experience conflict.
In practice, any sensible engineer will make his threads "fire and forget" - i.e. run, do some work, write some output, signal finished - as any time you need long running threads with persistent and detailed inter-thread communications, it is asking for trouble, like juggling with fireworks.
Threads are becoming ever more important, and yet there is not a single library which is worth a damn, i.e. is usable for the programmer, can get near full usage out of available hardware, and is easy to make safe and inspire confidence that after long times or heavy loads catastrophe does not occur.
Even when one has been very careful in the use of threads - and it can be impressive when it works - you will still end up with unsightly lock-ups of the main thread at times and an unresponsive application; I find this unacceptable even if it is merely for a few seconds.
C++ is a terrible language, and yet superior to everything else for general purpose, reasonably high performance, applications programming; the new language additions and the boost libraries might make it almost bearable some day; I do not see the point of Objective C, D, Java, C# - all just various variations of sugar frosting on a turd - as for pure-style old-skool C on its own, ... C is a systems programming language, a kind of high level assembler, which never should have been used in applications programming, where it has been a disaster; the language itself is blameless in all of this, the fault is due to corporate managers in the mid 1970s who found that they could hire C-knowledgeable graduates for a lot less than their previous programmers.
In almost all practical situations Python is the best language to use - for getting things done quickly with reasonable style and with the least aggravation, its probably the best choice - performance? - get yourself a profiler, find the bottleneck, and if you need to, rewrite it in C++ - chances are you won't need to.
The goals of StockWave will be achieved as promoted, but it is hard to say when - it will go out when its ready, which is to say when I find it acceptable; if I would not use it myself, I would not inflict it on anyone else.
StockWave is being user-tested at the moment - lots of bugs, blemishes and unsightly things are being found, but most of these are trivial to fix; about 5% are harder to find, but again fixable, and a small rump of behaviours which are neither errors nor bugs, but simply "the way things are".
StockWave is nearly finished, but there is no release date yet.
Having surveyed a great many other stock market and options trading applications, I think they are almost all, terrible; a rotten user experience with dubious algorithmics underneath, and ridiculously expensive. The various freeware applications are fairly trivial and crappy - draw a chart with a few TA indicators - but at least they are not a rip-off. There is thus still a need for StockWave, it was, is and remains a good idea.
The market trading game may be a bust anyway, at least for retail investors and even sophisticated individual private traders - the hedge funds and investment banks are front running everyone else, trading today is little more than a version of algorithmic robot wars - the ordinary investor coming into the game, with his non-direct market access and paying big spreads is there simply to be fleeced. We need a crackdown here - give everyone the same prices at the same time, and anyone who submits an order must be prepared to have it executed (perhaps a minimal lifetime of a couple of seconds), furthermore they must have the funds to cover the trade ... otherwise it's fraud. Fraud in the financial markets should be made illegal - in practice as well as theory; radical, crazy idea I know ...
The internet should have democratized access to realtime market data, but against all odds, the commercial feed providers maintain their cartel - they are a particular bugbear of mine as I sometimes find myself looking at their APIs, none of which are very pretty; do not complain about this as they think they are doing you a favour. Opentick was an interesting experiment which came to nothing, it would seem. Yahoo Finance seems to be the default supplier of market data to the world, until their leaders start to see it as providing a public service and thus a candidate for the chopping block. Google Finance looked interesting, but now is "deprecated". In the old days, in the days of analogue TV, there was broadcast along with it a data-channel known variously as Ceefax, Teletext, Videotext etc. depending on what country you were in - you could easily extract share prices from this, and so with a cheap TV tuner for your PC, you could get all the data you needed for a flat fee of £20 or so; now we've gone digital it is kaput - note that although there is now a "digital teletext" there does not seem to be any way to get the data out. So much for progress. And don't get me started on the people who sell you historical data, especially historical tick data ...
It would be nice if the online brokerages provided direct access APIs for third parties - but almost none seem interested in this, Interactive Brokers is thus the best and only game in town.
StockWave on release will be fully functional trialware - you get to play with it, the full thing for 60 days, say, then we will ask for your credit card. Unfortunately, it will be expensive by the standards of ordinary software - this will be done for a number of reasons; firstly, we don't see it as a mass market product anymore, its a niche high end piece for people who already know a lot about trading and are looking for something new to give them an "edge"; and secondly we wish to, initially at least, keep the number of paying customers down - yes! you read that right - in order that we can give them a more personal service, particularly with integrating our system into their current setups, feeds, brokerages, trading style; we would rather have a lesser number of satisfied customers than a howling mob baying for our blood, or asking us why things slow to a crawl when they do a massive monte carlo simulation on a 6 year old laptop while trying to play Call of Duty and running firefox with 20 tabs open ...
Modern GPUs, CUDA, OpenCL are very attractive - the idea of having a heavy duty workhorse already in your graphics card upon which to offload your number crunching, is beguiling - but the algorithms of StockWave dont seem to map onto the GPU easily; Knights Peak might be the answer we are waiting for, but I reckon it will be expensive, as will "compute" versions of graphics cards which can do double precision properly. Or perhaps the wondrous, all knowing, all powerful, "Cloud" will save us?
A mass market version of StockWave could be possible - in this case it would be a simple mobile phone app which sends a few parameters to a running StockWave instance ... which is on the Cloud; the application would be free, the user would pay for the cloud processing required to do his analysis, a small percentage of which would go to us; the user could then choose from a number of online brokerages offering him best prices, and once executed he gets regular position updates and warnings if so required.
The help files and documentation orignally written for StockWave (in 2006-7) were meant to be enlightening, honest, understandable and at times, amusing as well, but always essentially more truthful than the standard financial newspeak which is found on all "serious" money websites; this was generally well received and popular, an eye opener to most people, however I had my own misgivings that I was being too flippant, waspish and cynical, and hence would not be taken seriously, or that it would reflect badly on the serious algorithms and processing power of the product. It turns out, with regard to my "cynicism" I was not being cynical enough - not even close to it - for what followed has been the largest ongoing tsunami of so far unpunished financial crime in all history - it has taken people like Max Keiser and his report, plus "tyler durden" at zerohedge to give people some idea of what it is that really goes in the markets. Which all goes to show that when you think you are being "edgy" you can be a long way short of the truth!
All websites and applications should have a third party API, which is free and requires no registration - the web works as well as it does because so much of it is free - it lubricates the experience, paywalls and registrations are like tollbooths and barriers stodging things up. The grey-area activities of screenscraping and reverse engineering of url structure should be legitimised and even encouraged through the use of plain text only versions of websites, this in a roundabout way may finally allow semantic web ideas to penetrate; the semantic web is a good idea which won't work because it adds a cost to the website creator, plain versions of websites would allow the "costs of creating meaning" to be shifted back onto the website user, and then only if he wants it.
A lot of young people are trying to make a fast buck by doing essentially trivial things and hoping to have a big hit in the app store - so that by hook or by crook or by viral hype you blag 10million users, then facebook buys you out for a billion - the odds of this working out are not much better than winning the lottery and all you are doing is adding to the general noise level, the cacophony of crap which infuses the worlds media. Try instead to do something worthwhile, which usually means for some reason other than money; having said that, we all need to eat. The original goal of StockWave was educational in the broadest sense - it seemed to me that almost everyone was being ripped off, one way or another, by the financial services industry, and what they needed to do was to look after their own money making their own investment decisions ... and they would need software to do this, software better than currently available, which seemed to do little more than draw a few charts with some useless technical indicators, and to peddle both wrong and discredited theories about how markets work, as well as, of course, encouraging you into opening an account with some particular brokerage or money manager.
When you begin a project you should take care to use the best tools available, for example I choose not to use raw Win32 or MFC because ... they are terrible, absolutely awful; however, in a long running project you must be aware that what you are using may become outdated, deprecated or obsolete before you are finished. This presents a lot of problems when you try to get support - when you explain that something "doesn't work in version 3.4" the person on the other end will likely just say "but we're up to version 9.83 - you need to upgrade" - and that's it. Tough shit, mate, you're on your own! StockWave for example, is a reasonably handsome application and works fine under the latest Windows 7 64bit, but the window docking doesn't quite work properly - it never really did, and I'm not sure how to fix it - I could write a lot of hacked up code to check for docking and client rectangles, but it would likely be very brittle; I could write a docking manager component but that would be time consuming - some 3rd party components are available, but often the documentation is lacking. Note that the "migration report" generated by the latest compilers "auto-migration tool" was quite nightmarish to behold. The situation is further complicated by Microsoft's new operating system which seems very different from any previous version and seems intent on ridding us of the traditional desktop and with it the traditional desktop application.
Lastly, a proportion of the delays in this project must be laid squarely at Bethesda Studios, firstly for Fallout 3 and then more recently for Elder Scrolls V : Skyrim - do you people have any idea the damage you are causing to the economy? I used to do StockWave like you, but then I took an arrow in the knee ...

3 September 2010

Anyone who goes hillwalking / rambling / hiking will be aware of the phenomenon of the "false summit" - fixated on your goal you stride strongly towards a peak, almost at the top you notice another peak, significantly higher and further away than the one you have just achieved; psychologically this is like a kick in the ribs, not life threatening, but it takes the wind out of you. A series of false summits can make you feel like quitting though.

I haven't been updating the website as I had hoped my next update would be "StockWave 2.0 - released", alas I have been stuck in the "testing badlands" for weeks now. Note that the 3D data visualisation project is "finished" but untested - given the large amount of new material in this, I do not expect it to go in without a fight as there is a lot of scary Heath-Robinson-style algorithmics underlying it. On the StockWave side of things, there is good news and bad news; for example, it looks like the scripting language will work quite nicely and with the Python binding will allow a lot of nice things to happen for free (SWIG seems very powerful), for example, lots of very nice girls and boys have written and released Python libraries for almost everything under the sun, including trading - e.g. automated trading using Interactive Brokers, various datafeeds and webservices, data format converters, high level database tools and so on. This means the common question : "Why don't you do/support/have some feature/interface/indicator X Y Z"? has the answer : "do it in Python!" and you google for it. For example, if you want lots of technical indicators, you can use TALIB, which has a python interface, or if you want to code a "moving average crossover donchian channel breakout" trading system you get the code from somewhere and "python-ize" it.

KAGI and try before you buy - KAGI have a module called KRM which lets you embed an online shop into your application; when you're ready to buy you just do it from inside the application itself and all the secure credit card handling is done for you. This will make the new program easy to buy.

Extreme multithreading has been of mixed benefit - using one thread per datasource leads to occasional bursts of every high CPU utilisation; using worker threads within the advanced analyzers has led to some excellent speedups but also strange crashes (- probably threads trying to use too much memory). Advanced analyzers are still nowhere-near realtime - run times are going down from around 5 hours to 20 minutes, that's good but we are still left with the problem of the need to tweak our algorithms; it's frustrating to wait, but it's worse when what you wait for is not that good and you have to keep making adjustments. This frustration is increased because with the new, faster, multithreading it is impossible to view the "work in progress" of an analyzer (you have to trade speed for interactivity). A separate application designed for parameter tweaking of the advanced analyzers should be made - it will be much slower, but all the innards of the algorithms will be exposed and watch-able in realtime.

And another thing - I've been running Windows 7 64bit for a while now and would mostly sing its praises, but there's something I've noticed about it; when I am running old StockWave (which I do daily to keep price and news archives up to date), it regularly reports (few hours) that old StockWave has crashed. Sometimes if you ignore the crash form, it will continue on its merry way, but other times it becomes unresponsive. I've tried tweaking the compatibility options but the problem remains.

8 March 2010

I have been designing a new website for StockWave; this is of great vexation to me for several reasons. Firstly, web design ... is for arseholes!! (US translation : "assholes" - we like to keep the 'r' as it's a wee bit harsher.) You get some art-school halfwit with a ponytail, a man-bag and funky spectacles trying to bilk you for 5 grand for a dinky logo and a ten page website using his home-brewed and massively over-engineered content management system; and as for so-called "web programming" the coding standards you observe whenever you "view page source" in your browser are mostly, just terrible.

The solution to having to do "work only fit for arseholes" is to pay some (hopefully minor) arsehole to do it for you, having knocked him down a bit pricewise. This is what I did originally and came to regret it - any time you take your eye off the ball even for the best of reasons, like doing "real work", is a risk. It didn't really work out.

The current site is not that bad on the face of it, but a number of small things add up to a website that is simply not up to the job. First of all, it is fixed width, which on modern wide and high resolution monitors, does not look right. It does not use the space well, and the text is not easy to read. The fonts used are too small (one thing which is easy to fix at least), and the site is written in php; php is not of itself evil, but it seems to be the tool of choice for a lot of sloppy people - however since the site is almost totally static anyway, what was wrong with simple html? Visually, the site just looks too dark, like a Russian hackers warez download - every time I look at it I feel my credit card is about to be phished - it scares me, and it is my own website. It does not have a proper e-commerce buy online / license management/ credit card payment facility - the mail order system seems bizarre and antiquated. Overall it does not inspire confidence. There is also a mysql database on the back end - I don't know what this does and I don't know what it is for, for all I know my site could be spamming porn and vicodin ads to the nine vectors; why is it there?!

So I decided to do the redesign myself - my conclusion after a few days on it fulltime is that web design is a bit like driving a motor car, it looks like it might be difficult, but in the end, probably anyone can do it. I decided to use xhtml and css, with a 3 column liquid layout, this was apparently all the rage a while back but maybe not so much anymore - the point is, it works well when you have a lot of text on your site. Visually, I like the logo and also the simple, clean look of the current site's help file pages, which were added almost as an afterthought - so I will build on this. Fancy websites leave people confused, I think - they are a triumph of style over substances; too much "flash", literally; small fonts that are too hard to read and the usual corporate bullshit by the bucketload - "your confidence is our ambition" / "energise your experience" / "envisionise the team-goal thought-space"; what about "I am a plumber and I fix pipes" or "I sell software my software is good". Simplicity and clarity are much preferred.

One thing I have noticed about web people is that they tend to have quite narrow skill sets - they know how to do certain things, but only a certain way; personally I don't like to micromanage people or specify implementations, but ... my original designer, when I finally checked him out in greater detail, seemed to have produced a lot of sites which were almost structurally identical; he would change the colour scheme and build them all in the same way, like as if from a master template; nor had he actually ever built a site which had any e-commerce facility. There were further problems, he was, it seems "bipolar" (the current fashionable term for manic-depressive) and would go AWOL for long periods.

Another unnatural compromise was the splitting of the application into Basic and Advanced Editions - the true solution here is to use a trialware solution with a trusted payment provider (- looking at Kagi, Plimus and Avangate). The website will also have more demo videos, which will "ease the pain" for a lot of users - there are some complex 3 and 4 stage operations which are easy to do once you've seen them but hard to work out if you haven't, and with textual descriptions that are very long-winded.

The web design work is almost a bit of light relief (and more buzzwords for the CV) from the upcoming merge of the 3D work with StockWave itself, then the inevitable massive test and debug of all the new features (- I'm crapping meself at the thought!) But I want the site ready so when the new StockWave is finally ready it will simply be a case of pasting a link to the installer and switching it on.

A final, rather worrying thought, as the corporations move all their operations onto the web, Capitalism itself will be running on this all this "monkey code" - cut and pasted php, dodgy javascripts, newbie graduates first efforts, shockingly coded cruft, fragile legacy code ... there is even a website called "The Daily WTF" which catalogs this.

1 March 2010

The 3D visualisation project has progressed to the stage where we can draw an interesting range of primitives to represent different types of data. Getting the interactivity right seems to be a difficult problem - you really need datagloves (or other pointing devices) with a speech interface to make it all "natural", but that will come some day (hopefully the current fad in "touch" interfaces will spill over into the third dimension) - but right now we have to be content with mouse and keyboard - it seems OK, so far.

There are other problems to be aware of, like rendering very large or high-dimensional datasets, but these will be tackled as they arise - the techniques to deal with them are known; for example you usually wish to get an overview of a dataset, then search or filter on certain aspects, then perhaps drilling down to microscopic detail - you may have visualisations for companies, various aspects of the business model, then down to individual people and so on even into their personal data and characteristics (a name, a photo, their home address, their masters thesis, GPS data from their car ... potentially, at least), plus a forest of relational connectives between all of these things; this is potentially a huge dataset - you can't simply load it all at startup and render it as you go in a dumb and simplistic manner. (We will add level of detail hierarchies and k-d trees when the time comes ... don't worry, we know what to do!)

There is thus a 3D browser application that has been developed separately from StockWave - the hope is then to "drop" it into whatever datasets StockWave generates. As a first step it could be used to generate concept models from the news archives, or interactive timelines or show the webagent as it spiders between websites. It is very general and is designed to allow practically any dataset you want to be visualised interactively.

Getting this to work further opens up new areas for analysis, e.g. a news concept model will help to build (or test the feasibility of) a "predictive news model" which we can then incorporate into the monte carlo simulations as a new kind of "noise" source.

And still there's more - the 3D stuff now gets put aside for a while and the StockWave project brought up to speed; there is much in this that has not really been tested in earnest - the datacapture and monte carlo algorithms were re-written to spawn more threads to use multicore systems (and to scale up as large or as fast as your hardware will allow) and the scripting should be a very exciting addition; it opens up the system to the full "algorithmic trading systems" people as well as allowing the less tech-savvy user the ability to "one-click" his way through many things; in retrospect, StockWave was a tricky application to use and could be off-putting.

1 September 2009 Processors and Benchmarks

One of the worrying things we found when creating StockWave was the length of time it took for the algorithms to run - at the time there were no multi-processors on the desktop and so it was a tough sell, i.e. the prospect of having the user wait some hours before getting a result; havng said that, the program had bags of other stuff which made it useful anyway, but the centrepiece of the analysis toolkit ... simply took too long.

Since then technology has caught up with us and our methods mostly take well to multiple threads and processors; once 8 or 16 cores are standard on the desktop I think this could lead to a revolution in online trading, vast new territories of trading possibilities could open up to the retail investor.

(To any of you that are curious - the advanced analyzers do NOT map onto forms which can run on the GPU, not even with CUDA, however the GPU will do basic random walks very quickly.)

The point I'm really getting at is - for us, getting what you want out of StockWave depends on what number crunching you've got available, for us, there is never too much, and corrrespondingly you need to be able to get the best bang for your buck computer-wise - the very top processors often provide only slighter better performance than their second-bests while being ridiculously more expensive; Intel's "Extreme" series processors come to mind here - you really have to be a more-money-than-sense stupid prick to buy one of them.

Actual performance results often produce surprises here - the number one rule about performance and optimisation, is to actually measure it in the first place; some people forget this (- the true performance determinators / bottlenecks are often quite unexpected). I have been using a benchmarking program - free 30 day trial - called Performance Test - it uses a large collection of real world benchmarks and lets you access their database of results for comparison; it gives breakdown of benchmarks for CPU / memory / graphics / disk / CD plus an overall figure. Note that the one most interesting to us is CPU, especially floating point performance.

The results are quite surprising in many ways - I present these as a list of miscellaneous observations without wishing to draw any premature conclusions -

The most basic observation is that the usual expected variational dependencies do exist, but are often much weaker than you might think they should be; but also, apparently similar systems with the same specifications - same processor, ram, graphics card - can vary quite a bit in performance among themselves! The practical repercussions are obvious here, both potentially useful and worrying at the same time - you could pay modestly for an apparent middling system which is in reality a real flying machine, or, pay well over the odds for a relative dud. With a few hours tuition and watching it done a couple of times, we could all, every one of us, build our own PC to our precise requirements, there's no real excuse for it except laziness - you want it quiet, or you want it fast, or you want it for games, or you want it just for web and email? Build it yourself!
Intel's chips fall into two main classes - "Nehalem", branded as i7, and "Penryn", branded as Core 2 Quad; there is a range of models, differentiated by clockspeed and cache size; i920 940 965 975 and Q8300 8400 9300 9400 9450 9550 9650. I will ignore older chips and anything with less than 4 cores.
Dear old AMD's best new chips are only getting to grips with Intel's older line - the Penryns, so I wont't bother with them; like lots of people I have a soft spot for AMD, they always gave good value and often beat Intel in performance, but now a gap has opened up - however, and this is very important to remember, AMD's chips are the best bang for the buck for almost all PC users; weirdos like me who want to do monte carlo simulations and manipulate large datasets are very much a minority.
The Nehalem class beats the Penryn class by significant margin, but there is a weak dependence within each class - "inferior" processors can often exceed "better" ones, or be easily overclocked to do so.
There is a weak dependence on clockspeed within classes - if you think the 3.2 GHz model will just "melt" the 2.4 GHz chip, it probably will, but not always, and the difference may be trivial anyway; it's a pity the faster chip cost 3 times the cheaper one!
The actual clockspeeds of each chip can vary quite a bit from what is on the box - e.g. my 2.66 is only 2.48.
Overclocking can make a huge difference - or almost non at all; this is one aspect that you would expect to find a nice linear relationship, but there isn't. Note that Intel seems to put in a lot of safety room for what it clocks it's chips at - a 30% overclock without much tomfoolery is relatively straighforward; Nehalems can go to 4GHz or better, and the Penryns to about 3.2GHz.
Higher-end Penryns can be overclocked into low-end Nehalem country, and
Low-end Nehalems can be overclocked to equal their top-end rivals.
Every processor is subtly different, a unique individual - some just come out superstars, while others are a bit "retarded", sorry "special", but not in a good way.
There is a weak dependence in going from 4G to 8G RAM; whether large amounts,16,24,48 give you anything is unknown; what is sure is that less than 2G is going to hurt you - this is the minimum amount for a modern system.
There is a weak variation between Windows XP, Vista and Windows 7 operating systems when using the same number of bits, but
The biggest surprise I found is that the largest single performance increment seems to be in moving from a 32bit operating system to a 64bit one; the XP, Vista, 7 dependence is tiny but from 32bit to 64bit is big. Naively you can see this is correct - you are shifting bigger chunks of data, it is more efficient hence faster, with less swapping to the hard drive etc.
ASUS seem to make good motherboards - they are well represented among the fastest systems in the database.
Dell systems are rather average performers - it's easy to buy from them and their support is great, but their PCs aren't flying machines.
The current value-for-money chip has got to be the i920 or Q9550, although the new i5 class may well muddy this. So, in conclusion the "sweetspot" would be a good Penryn (since the motherboard and memory will be cheaper), any 64bit OS and 8G RAM, with an ASUS motherboard; this is probably the cheapest, really fast machine you can build. The very fastest OS is probably 64bit XP Professional SP3, though I don't think you can get hold of it very easily.

BTW, if you like this sort of thing, then the AnandTech and TechReport websites are where you should be hanging out - for them it's a full time job.

Taking a wider view for a moment, I must say that for ordinary folks, performance considerations are totally pointless, and why do some of them overclock? - it's like boy racers modifying their cheap, crappy cars into ugly, lower, louder, cheap crappy cars. If all you need to do is to websurf and email then any PC will do the job well - it's the speed of your broadband that will be the limiting factor. And if you are a big gamer wanting to enjoy the latest eyecandy, then you should be spending your money on the baddest graphics card you can afford, with a nice large, wide monitor.

Looking forward to my next trade-up, I really like the look of the Dell T7500 workstation with the dual sockets, the tech geek in me drools slightly over those xeons - 8 cores with hyperthreading ... mmmm (sounds like homer simpson contemplating a hot dog) ... on the other hand the cynic and the skeptic in me, who are disgustingly, almost unerringly correct in their every judgement and prognostication kicks in here; aren't "Xeons" just a marketing brand-name, a ploy to get corporate types to pay a premium for "server/workstation class" chips? Consider the long running outrageous scam that operates in the graphics card market for "certified" cards like Quadro and FireGL - isn't it just the same? Let's check the figures on the database - dual xeon system at 3.2GHz is around 16000 on the CPU mark, my humble Q9450 is 3500 (though some systems have been scored about 5000) - let's be generous and say the top of the line system will do 5 times the work of my current one, but costs, alas about 10 times the other one! In terms of doing StockWave number crunching, it may reduce an analysis runtime down from half an hour down to 5 minutes - but that could be worth quite a bit of money as while you were waiting the extra time for your full analysis, the prices could have moved a lot against you. Which is all very interesting - these "value for money" questions have to be settled from a global view point; if it's the difference between profit and loss, the very fastest chip may pay for itself, easily!

8 June 2009

Numbers, even after much crunching, don't mean much - that's why we need graphics or rather visualisation to help us "understand"; the heatmaps in StockWave show time dependent probability distributions for where the share price might go, a lot of numbers condensed to a little picture that tells you what you need; the news-chart and event stream shows stories alongside share price moves and also gives an indicator of the news sentiment at the time; these straightforward visualisations seem to do the trick, but what happens when you've got some arbitrary, crazy, data-set that you might hope has some useful information buried in it - what happens when there is no clear way to look at this data, no obvious approach?

The new feature we are working on is a generalised solution to the data visualisation problem - for any old weird set of datatypes, in any number of dimensions; to create a 3D picture which generates itself automatically from anything you stick into it and then lets you explore it, tune it, morph it into something which lets you extract useful information. Then once you have something which you like, graft onto it the standard types of algorithms that will work on particular aspects of it - stick a kohonen map here, do a k-means over there, run an SQL query on this, and tie it together into a hidden markov model; whatever you want, totally open.

So, obviously we have to do 3D graphics - this is just a matter of being clear what you are doing, refreshing your vector and matrix maths, and at all times remembering what direction is up. Still, your debugging sessions can be very invigorating.

Graphics cards these days are incredible beasts, capable of incredible fill rates and polygon counts, but all they are really doing is drawing texture-mapped triangles - that's all you really need to know how to do.

One has to make a choice about what to use as a rendering library; for some time now games programmers have been using DirectX / Direct3D almost exclusively, but their concerns - to get the latest hardware shader special effect working - are not ours, which is a good thing, as I think DirectX is based on COM (- the underlying design I mean), which is not a good thing. The other way to go is OpenGL, which we have some previous experience of, so that's that then!

On the face of it, OpenGL seems like the best choice for us - it is open, and well-supported, and all those touchy-feely good-time things and it seems quite straightforward if you've done a computer graphics class. But it is very low-level, and frankly, I hate it. What you have to remember that it is designed as a State Machine; this project and most large projects these days use object-oriented design principles - it just doesn't go together naturally, so not only do we have to use OpenGL but we have to create an "object-ified" wrapper for it.

Of course there are plenty of 3D engines out there - too many, with every variety of sophistication - but none of them seem quite like what we need, and when you use these you often have to include practically "the kitchen sink", and still need to do a bit of hacking to make it fit.

State machines are annoying to use when in the "object" mind-set as you must never forget that the order of operations matters, and when you set a switch it stays on until you reset it - it does not reset once you leave the current scope; if you have a black screen, or funny colours which change unpredictably, or objects which translate oddly or rotate about odd vectors, textures which disappear, or the wrong textures on the wrong objects, then it's all because you've probably done something and forgotten to undo it, e.g. that classic of Opengl newbies, forgetting to pop the matrix stack.

The documentation is not as good as you might think - the "Red Book" (now availabe for free online) contains lots of examples, but they contain very -simple- examples only. There is no advice on design, on how to use OpenGL within a larger project - no "how to put it all together", certainly nothing on making OpenGL into an object design.

Obscure problems, things you would never imagine, and answers to which can only be found on the newsgroups - are quite common, e.g. some commands don't work inside others, or need to be done in a certain order (!), otherwise they don't work; this can lead to mammoth debugging sessions with oodles of trial and error - this kind of grind makes programmers very depressed.

I am proposing a new marketing slogan to the Open Standards Group people -

"OpenGL - not quite as shit as DirectX"

On a completely different matter, I seem to have lost access to the company email accounts - Doh! - I will set-up a gmail or yahoo alternative as soon as I can, not that there's much to really talk about in the immediate short term; scripting and Python is in, which I think will really open up the system for people, but there is loads of testing to be done, as soon as this 3D stuff is ready.

Update on the hardware problems - if you have had problems with the firmware bug on your Seagate Barracudas, Seagate will fix them free of charge; they won't exactly track you down and do it for you, but if you logon to their i365 site and check the serial nos, you can open up a recovery case and they will fix the drives and ship them back to you in about a week. For free!

22 March 2009

I have just noticed something interesting about the news event analysis tools - they seem to work better than expected! In the documentation there is some description of them, plus the usual caveats about it being an experimental feature, well, some way down the road, and with quite a lot of news stories in the database, I have been creating news event streams, i.e. "market sentiment" indicators for various stocks. For the FTSE 100 it seems to give a good leading - yes, a leading - indicator for trend direction; it runs about a month ahead, but note that it is changeable and is prone to whipsawing around, neither is the news model in any way predictive (- more on that later). Still, that's not bad for a "side dish".

I'm going to try and push on with the "super web agent" next - this is a web agent which can, while it is spidering, suck out all the goodness - names, people, companies, organisations, events, states, and show it graphically. This will have a lot of practical use I think - being a great user of the web it is very frustrating just how much crap there is to dig through and how low the signal to noise ratio is.

The fundamentals database can go on the back-burner for now; fundamentals are fictions anyway as the latest round of swindling has shown - all these companies going down the stank had their accounts signed-off true and proper, and their debt all rated AAA; data as corrupted as this cannot be data-mined for anything worthwhile - it's like building a house on quicksand. Companies go to great lengths to create useful fictions about themselves - so it must be the side channels we need to examine to get our "information leakage" and any signs of anomalous behaviour - which is better suited to the web agent.

17 March 2009

At the risk of this becoming a blog, a literary form I detest (- if you can actually write, then go write a bloody book) I need to vent and blow-off a little. Firstly, some product reviews -

Fancy new PC has had two Seagate Barracuda 7200.11 drives fail on me - one after 1000 hours the other after 700; given a claimed MTBF of 750000 hours this is "impossible". Dell support is not up to much, "next business day" ends up as "one week later"; talking to them is like conversing with polite but rather slow 12 year olds (- who can be scary smart at times!); you need to tell them 3 times to do something, then they phone you up to confirm it; not too quick at answering emails, and not too good at giving direct answers. While waiting, I just ordered a new 1T Hitachi drive from Misco, got it next day, and was back up and running in 4 hours. How hard is that?

The hard drive failures made me look into disk cloning solutions; Acronis True Image, which is incredibly highly rated over the web-o-sphere (- by the company's shills?) is the most useless backup and cloning software there is. Don't take it even if offered it for free. (I returned mine for a refund.)

Opentick has gone and done what I guessed they might do. Shame I've now got a ton of their API calls in my codebase, but who knows, maybe one day they'll come back. Taking a wider view, looking back, it simply didn't make sense - how could they make money on this, if it was not for some other reason? The reason seems to be that the whole enterprise was a large-scale beta test to iron out the bugs in their real product, Xasax. Question: will all that dormant Opentick code be able to interface with Xasax?

How StockWave will support datafeeds in the future is thus - not at all; really I just can't be bothered. OK, actually we will, but here's how - you get the standalone feed reader application from your supplier, then you phone their helpdesk and ask how to export data in CSV format, if they get suspicious, tell them you want to put it in an Excel spreadsheet; they send you a script which can do this for you. Now you email us, and we show you how to create a "FilePriceSource" - this will read in the exported CSV text files on a periodic basis. Voila! And no messing around with their APIS, DLLs and signing NDAs. Our hands are kept clean, and a hundred hours or so of developer time is saved for something worthwhile.

My "can't be bothered" remarks point to something of a deeper malaise; fact is folks, programming is ... shit. Yes, honestly, it is. Not a good career, a refuge for second rate intellects who think they are the cream of the crop; a lot of social inepts and aspberger freaks. It is only bearable, if you are working on your own project, which has some potentially deep or important innovation. Software such as this cannot be effectively project managed - it is similar to blue-sky research, and has payoffs, if any, which are highly nonlinear and unpredictable. It's a tough gig just to keep going.

There is a highly rated programming blog which is obsessed with the idea of "rock star programmers" - this is a risible concept; if you are working in a corporate environment, for someone else, then, by definition, you cannot be a "rock star programmer". You are a maintenance programmer with a nice-sounding job title. No genuine "rock star programmer" would ever call himself such, wish to be, or seek to be known as such; they are also pretty-much un-hireable as they simply will not do other people's tedious shit.

(I am not a rock-star programmer, I am an independent scientist who does a bit of "karoake" programming on the side; programming is not creation, it is implementation, something the hired help does. And when the hired help is not available, you have to roll your own sleeves up.)

Getting the latest StockWave out has been hard as there is a lot of "tedious shit" to do - no big issues, no design innovations, no novel algorithms, just fiddly crap. And herein lies the real problem - we have another project on the go, it is very open-ended and wide-ranging (- it will feed back into StockWave eventually) in the general area of information processing, and equally importantly, visualisation; a lot of effort has been spent on this, for no visible benefit as yet. But the other project is a lot more interesting at the moment as StockWave is quite mature.

Must stop this, starting to moan like my old Aunt Nelly.

5 March 2009

Normally I don't bother much with what other people are doing; let them - so what if they say this, or claim that, but recently I've been looking at a number of websites for trading software companies for comparison as our site needs a bit of a revamp; it's too dark and forbidding. Obviously these guys are all furiously hype-ing their wares, as is their right, but some of the claims they make are really annoying, as the gap between their claims and the reality is often a mile wide. Naming no names, to protect the guilty.

Practically everyone claims to offer "advanced analytics" - what does this actually mean? For example -

Do you have monte carlo simulation?
Do you have machine learning algorithms?
Do you use signal processing?
Do you have news event analysis?
Do you have specialised web agents to seach the web for you?

No, none of the above - what you really get is a few dozen technical indicators drawn on your chart; this is not "advanced analytics" - this is technicolour voodoo for the self-brainwashed hopeless optimist.

"Datamining" is another popular buzzword they like to use -

Do you get advanced visualisations of high dimensional datasets?
Do you have pattern extraction techniques to identify signals of good and bad companies?

No, they have a very basic scanner which can pick out certain stock criterion like market capitalisation, PE ratios and volatility; this is not datamining it's just an arbitrary pre-selection.

Everybody is into "advanced trading sysems" as well, but - is this a customised probability model which can search for specialised and novel trading opportuntities?

Again, no - it's a formula editor with a backtest facility; this is not an "advanced trading system" - it's simplistic guesswork and a "wet finger in the air".

An "easy-to-use programming interface" - well, you just have to have one of those now, don't you!

So, does this API use one of the most popular and powerful, loosely-typed scripting languages, which most people can get up to speed with in under a week, and is freely available with many supporting libraries?

Er, yet again no, not really - it is instead our own custom language which we have invented all by ourselves just the way we like it - with a mass of idiosycrasies and unfamilair syntax, all inspired by the company founder's notions of just what good coding style really is (- because he was a real cool cobol coder-dude or was a visual basic "guru" in his pomp); also comes with a boatload of bugs and library interdependencies to test you with, and all guaranteeing in the end, that no one but us will ever use it - that way we've got you forever!

Somebody should get these guys under the Trade Descriptions Act.

not-really-sure-what-day-it-is, January 2009

Happy New Year to everyone! And Merry Christmas before that, as well; our turkey was moist, and our kapitan, our glorious leader, well-pleased with his gift, a pair of Oakley sunglasses to hide his eyes while he tortures us to higher productivity at Stalag StockWave. (- that's enough of that! Mocking sarcasm, 5 seconds into the New Year. Hmm.)

What's been going on, in no particular order, or even logical structure -

Teletext is I think by now mostly kaput everywhere, goodbye little friend! The code to use it is still in there, so it will still work if the service is available and you have WinTV/VTPlus installed, but please don't send us emails asking why things have stopped working - it's the governments doing, not ours. Write to your MP - try sending him anonymous threats - it's always worked for us!

Have added a treeview component to the main form as well as the usual scrollable grid; this took a little bit of time to get working, and it does look OK, but it does make the user interface busier, and on reflection, it adds absolutely nothing that the user really needs; make a mental note here - do not add pointless eye-candy which just "looks kewl" without a good functional reason; remember this the next time someone wants to be able to dock all the application sub-forms into the main form (- what a bloody mess that would be! If you're a "docking junkie" then try Eclipse.)

Being megalomaniacs at heart we'd like everyone to use our program - and for everyone to use "StockWave" as a proxy phrase for "trading software" just like the way, back in the day, "Hoover" was synonymous with "vacuum cleaner"; to have a chance at this you have to cater for the needs of folks in "other countries" of course, now while China and Japan would be too much of a stretch, we did try to get India and Australia covered; not sure it actually worked though, probably down to glitches in timezone handling code and rules for daylight saving time. Now, you might think we're idiots for having trouble here - surely you just read it from the system clock, and take a bit away or add a bit on, well, to use a wonderfully crazy phrase I heard only recently : "in theory, theory and practice are the same, but in practice, they are different". Anyone actually living out there, genuinely trying to use StockWave, give us some feedback please. Obviously there a lot of potential users out there for us - intelligent, well-educated, English-speaking folks, ... and Australians too, and we would like to reach them.

Finally, moving to proper source control, which should allow us to be a bit faster and more aggressive in our work. Note to young developers out there - winzip with a timestamp is not source control.

An improved form for displaying option chains has been added - it can also download these from Yahoo, furthermore the option prices can be shown on the stockchart as well which is nice. We have also created an on-chart visualisation of level 2 data - this lets you see individual orders and fills "dance around" in realtime, alas we don't have a level 2 feed to test it with! (God, we're so stingy ...)

WTF is happening with opentick, we ask ourselves? When opentick started to offer free non-delayed streaming data we thought "Wow, let's support that - just what the world is waiting for"; downloading their libraries and API we were further thrilled to see that the programming interface is a thing of clarity and simplicity, which is not a small issue; other providers' APIs typically induce mental torment, they also charge you an arm-and-a-leg, and also make you sign ridiculous NDAs to stop you "stealing their trade secrets" (- like what?! - reading data through a socket in a thread ... no one's ever seen that before - better patent it quick!); generally they act like they're doing you a huge favour. Supporting lots of feed providers means writing the "same" code over and over again each time dealng with the often bizarre and peculiar idiosyncrasies of their APIs, then keeping up-to-date with their frequent and often arbitrary changes. This leads to major code bloat, is a maintenance nightmare and leads to significant customer problems as the public can get -justifiably- very irate if their feed goes haywire, which is always seen as the application developers fault and not the feed provider - since "it worked OK on my other software"! Then they log on to some traders website and trash you mercilessly. But then Opentick came along - seeming like a breath of fresh air, a clean design and free apart from exchange fees (- a few bucks!) - it seemed like a new dawn.

Anway, we quickly knocked up a standalone grid-view based downloader, and it worked really well, we even put flashing red and green backgrounds for price direction when updating (- woo hoo!); this was quite groovy in itself, but it never got put in to StockWave down to what I would call "niggles" - little bugs we couldn't get to the bottom of; rather than get sidetracked by a side-issue like this we got onto other more important things (- sometimes in these situations, it resolves itself, or the solution suddenly becomes obvious). This was well over a year ago; we've only recently integrated the opentick downloader into StockWave, only to find it doesn't work at all now - we logon, get a few messages, but don't receive any ticks, then the connection breaks after a while (- yes, the test account we are using is still active ... thanks for the suggestion!) Have contacted them about this, but there is no apparent solution.

Opentick are owned by a company called Xasax, their prime interest is in really high quality, high frequency fault resistant datafeeds and order routing (- impressive technology using custom hardware and specialised chips, i.e. FPGAs) - it's designed for the next frontier, the remaining undiscovered country of trading - high-frequency algorithmic trading; you are reading data at the sub-millisecond resolution, doing some analytics, then maybe making dozens or hundreds of trading decisions on this per second; this is way beyond any human decision making. There are quite a few companies investing in very expensive, high-bandwidth "plumbing" hoping to give themselves an edge; at very high frequencies you are hoping that there are numerous inefficiencies which you can exploit with some relatively straightforward algorithms, but if you can also gain an edge by getting your price quotes a fraction of a second sooner than the next guy, or can find a price improvement at execution time ... that also helps. A lot.

Xasax product does not seem to be available yet, but don't hold your breath little "retail investor" because it will be stratospherically beyond your price range; opentick stopped taking on customers a while back, and so it might all have been just a hobby for them - a kind of "big beta" experiment to act as a testbed for the "real deal" which people are going to be paying for; presumably once the big-paying real customers arrive, opentick might be allowed to wither on the vine - which would be a real shame. Let's hope this is not the case.

A few words about this high frequency business, and why the retail investor shouldn't worry about it (- too much ...) - high frequency is not and will probably never be available to the ordinary investor - this is big boys' toys for big boys' games; it's just an arms race in trading technology infrastructure, which will probably end up in a stalemate. Market regulators ultimately must ensure that the playing field is (somewhat) level. Note also that at such high frequencies, you can't do much in the way of analysis, certainly not the deeply multi-faceted monte-carlo based processing that StockWave does, it's simple stuff running really really fast fast FAST. I would rather make fewer trading decisions based on a deeper analysis than lots of fast ones based on some simplistic scalping technique which to work needs to have a speed advantage over everyone else. And for the trading to become fully automated - I just don't like, the human being should never be taken out of the decision making loop; human beings have "common sense" and intuition, they also are implicitly aware of context, and are good "bullshit detectors". Proponents of high frequency would argue that at short enough timescales things will become simpler and that the "fusionist" philosophy of StockWave is not required.

Multiple core processors are now fully supported in StockWave - we always had extensive use of threads, but now we can spawn even more "worker threads" depending on how many cores there are available - this allows us to take full advantage of the hardware. We've got a new machine to test this, a Dell XPS430 - a very sweet machine, at a good price; it's fast, very fast, even though it's not even the latest Core i7 processor, but "only" a Core2 Quad Q9450; a full StockWave build takes only a few minutes. Other observations of potential interest; Windows Vista (Home Premium) seems, ugh ... "quite good", I hate to say (- I'd have had XP installed by default had I the option), still it would be madness to install it as an upgrade on an old PC; if you are a trader than 23" or better widescreen single monitor set-up should be good enough for you - multiple monitor configurations will go the way of the dodo. Anyway, the main thing is that our monte carlo goes like the clappers - it's great to see the algorithm start, ramp up the cores to maximum, then crash back to idle.

Python and scripting - getting there, but there's something I need to do with a library called SWIG.

There loads of other stuff still to be tested and integrated.

Have also been thinking about adding more content to the website and help files - the glossary in particular has grown into a collection of mini-essays, originally it was conceived as an antidote to the usual "explain it but tell you nothing" guides you often come across, hopefully delivering its insights in the guise of an acerbic one-liner, but some things need a bit more explaining, and some things shouldn't really be joked about, even if your humour is blacker than the inside of Satan's coal scuttle (- which ours is). And since everything affects everything else, we need to look at wider issues - the danger here is descending into "punditry", i.e. lazy, opinionated, self-serving wind-baggery (- think of all these experts currently being wheeled out to explain the complex details and reasons behind the ongoing market turmoils - a year ago most of them were probably saying the "fundamentals of the world economy are sound", and we should all just buy, buy, buy. Arseholes.)

Despite being locked in a lonely garret we are in conversation with users from all over the world; we get some truly interesting emails from some very intelligent folks and also some hilarious bollocks from real crackpots - both types of conversation are worthy of a wider audience, so they might get put on the website.

1 June 2008 Oh dear me, is that the time ... ?? What have we been doing!

Apologies for the current tardiness with regards to our email correspondence - work comes first but we're trying to respond to the backlog. Please be patient.

This is just an update for all our avid fans out there especially the Singaporean who looked at almost every single page on our site the other day - "big-up" to you (- just a wee hint though : the freebie download has all the documentation already with it!)

Interesting times re: the (various) market turmoils - we hope that no one got badly burned out there, still has a house, and money for the meter; of course we would, rather arrogantly, hope that having read some of our screeds you may have been a little forewarned, and therefore cautious in your dealings - the markets are intrinsically unstable and will fall apart, irregularly, every ten years or so, but only the big guys get the bailouts, so "Little Man" be-ware; and note also that this situation will continue indefinitely, or until some form of socialism is adopted in the major western economies, staring with the USA ...

(fits of giggles, rolling around laughing, biting the carpet, tears rolling down cheeks)

- pardon us, but we do like our little jokes; the facts are we're all stuck with capitalism until we die ... that's just the way it is (- of course, all of these corporate and banking bailouts are a kind of "welfare-state socialism for the rich", but that's not what we're really talking about.)

So where are we, exactly?

Technically, we've been laying the foundations of "future plans", basically it's coming together, and crucially, everything planned is do-able - and will be done, eventually. Specifically -

Opentick support will be in - but there no plans to support any other feeds; we became physically ill looking at the API documentation for some of the most popular feed providers, and our teams of lawyers fell asleep while reading their various NDAs.
Scripting with Python. Yup.
Moving the underlying data management to a database, probably PostgreSQL, but that will be left open, architecturally speaking.
3D visualisation stuff ... "getting there"
in particular, the confusing mess that was the TradeCreator form has been cleaned up, becoming "almost usable"; in a way, this was the crux of the application - to find custom, specifically-crafted, probabilistically-optimised trading opportunities - the fact it was so hard to use was a little embarrassing.

Commercially, we are going for a change of tack; getting rid of "basic edition" and "advanced edition" - never liked all of that nonsense, same goes for version numbers, 1.2,1.42,1.7833537.839393 build 28782738237 - oh dear lord almighty ... give it a rest!

There is only one product, it is called StockWave -

it is not free anymore, but
it has a generous trial period, and
it is very inexpensive if you want to buy it

There will be an "uber-StockWave", not generally available to the public, which will be our own research and trading/testing platform; it will contain things which are unproven, untested, hard-to-use, or computationally so demanding as to be outside the utility of the general public; we might sell you this version if you really want it, but it will be expensive, and we won't be supporting it; leading edge, or bleeding edge, it's experts-only with fat wallets.

Note that StockWave will be almost exactly-the-same as "uber-StockWave" except for the things which you probably can't use effectively anway, i.e. - it will not be "hobbled" to the extent that Basic Edition was compared with Advanced Edition.

There is still a planned "big test" to do - the good news is we don't need investors anymore as now, e.g. an 8-core machine can do an advanced analyzer in about 10 minutes! We thought we needed £50K+ and 4 months, but we can probably get by with £10K and 6 months; it has also been brought to our attention that "cloud computing" is now available with the (back-of-an-envelope calculation) capability to do an advanced study in 2 minutes at a cost of 17p! Which is something to think about ...

Website will be updated - the lack of proper online shop is, really, a bit of an embarrassment, and the use of mail order makes us seem like a bunch of scammers (- BTW your legal consumer protections are probably at least as strong with mail order as with online credit card payments, at least in the UK.)

(hi ho hi ho ...)

20 January 2008 Garbage In, Garbage Out; one of the good features of StockWave is the free high frequency historical price data - to get this we have to aggregate it ourselves from multiple sources, alas, it has been pointed out of late that the data is getting a bit ropey due to mismatches among the end of day and intra-day stuff - this is due to stock splits, reverse stock splits, various adjustments, bad reads, etc and so on. Look at ANTO and BARC for some chart-silliness.

There are tools within the program to maintain data integrity, and we could tell you how to go about fixing things when you come across a crazy chart but rather than bore or alienate our users ... we're going to just fix it ourselves.

To get the best available data when you open a chart go to the first drop down menu and 'download intra-day prices from web'; this will be the best available but if it still looks crazy then email us.

Since we are going through this lot by hand it might take while - 1.5 was to have the nifty and opentick feed support plus updated stocks lists - but we shall have to fix this first. When we're done we'll release a 1.42 version in the interim.

10 January 2008 Analogue signal turn-off; the UK is turning off the 'normal' TV signal fairly soon, which means that it will no longer be possible to get share prices (cheaply!) from teletext, thus this feature will be removed. We could investigate the possibility of looking at the data services provided on digital TV, but we aren't going to bother, as it seems almost all users are relying on Yahoo for streaming prices and our own backfill service. Ho hum - it was a nice, cheap, facility while it lasted. If anyone thinks getting data from DigiText is a good idea we might reconsider, but in the meantime, we'll be working on other stuff.

1 January 2008

StockWave Software Ltd invites approaches from investors and potential business partners.

StockWave Software is a small software company based in the west of Scotland; it is employee-owned with no debt; it has developed a highly innovative product for stock market data analysis using a synthesis of advanced scientific techniques. (The designer of StockWave holds advanced degrees from respected universities and has extensive work experience in high-tech engineering and software development.)

Our original intention was to provide scientific tools for the mass market of private investors but up against a mature market dominated by data-feed providers' packages and brokerage platforms, our success has been limited; a further problem for us is that our target customer market, the retail investor, is wedded to "Classical Technical Analysis" which he seems loathe to give up.

Commercial realities are that marketing to the masses is prohibitively expensive requiring a dedicated full-time sales team plus support staff, trainers, materials and advertising in print and web; although our - relatively feature-poor, freeware Basic Edition has found approximately 10000 users, this has not proved to be a bridge to Advanced Edition (- our full featured product) - the transfer rate is too small to be viable.

A different approach is therefore needed as alas it seems we cannot reach the "man in the street" (- please note that this "different approach" will not involve altering our product to meet the whims of dyed-in-the-wool "technicians") - we intend now to market StockWave as a niche software product to various financial experts and professionals such as quants, analysts, fund managers and professional traders. To be convincing to such skeptical and highly critical persons, we need to do a large-scale, statistically significant test; 4-6 months to map the sweet-spot of our advanced algorithms, and another 4-6 months to trade a typical portfolio.

Investment is thus required to purchase the necessary hardware to conduct our tests.

What we offer - appropriately, for what is, after all, a derivatives analysis package - is a Call Option, giving the holder the right to buy a large equity stake in StockWave Software Ltd for a very low price at a future date. This call option will be 'European-style' with an expiry date one month after the test results have been presented; the option holder will get a first, and exclusive, look at these results.

The premium paid for the call option will be used to purchase hardware; since we have a rough idea of what we need, and the target timescale for doing this study, we do have a ballpark figure in mind for the sum required - prospective investors should be able to work this out themselves, but we shall not mention any figure explicitly. We should point out that the figure required will be a modest amount to anyone we would see as a viable investor. Any early bid which significantly exceeds our reserve will be accepted as we want to get started as soon as possible - otherwise there will be an auction.

Further Notes

StockWave may contain patentable technologies, although we have made no study of this, we are confident that much of what has been done is innovative and unique.

For the above reason we cannot go into too much detail in explaining our ideas to prospective investors - we mention (some of) the ingredients, but not the recipe itself; having said that, our documentation and help files already contain as much information as we are going to provide to anyone - we would therefore encourage any potential investor to seek independent technical experts who can make some reasonable critical assessment of our approach.

The true value of our software may not be to private investors with their relatively limited computational resources but to, e.g. hedge funds or other large organisations interested in high frequency trading, statistical or event based arbitrage; StockWave is already multi-threaded and is scalable.

Advanced Edition may be removed from sale for the duration of the testing period, if so requested.

31 December 2007 This fruits of this year's technical development and new features will be drip-fed into StockWave over the coming months on a bi-monthly cycle going up through versions 1.2, 1.3, and 1.4. A 2.0 release will be made 6 months later. Advanced users will be afforded free uprades through 1.2 to 1.4; 2.0 will be a new product requiring a separate license, although existing users will have 50% discount.

19 December 2007 Merry Christmas and Happy New Year! We're shutting down over the holiday period, so see you in 2008.

29 October 2007 Basic Edition 1.41 released - this is a minor bugfix version.

28 September 2007 6000 users of Basic Edition.

27 August 2007 We've fallen a bit behind on our intended release schedule for Basic Edition 1.5 and Advanced Edition 1.2 - this is because everyone is just back from/on/just about to go on, holiday and those that are here want to work on the groovy stuff outlined in the Future Plans section of the site. We shall be cracking-the-whip soon though.

Monday 16 April Get the updated datasources from here; unzip the file to your install directory, your old datasources will be over-written. Any problems, send us an email.

Friday 13 April 2007 Black Friday Fun! Yahoo seem to have changed their time formats, outputting everything in Eastern US time; fix applied locally - updated datasource files will be available for download on Monday; backfill will be available as usual. This brings us neatly onto discussion of Basic Edition 1.5 - we have decided to support the OpenTick datafeed; if you don't know what this is, it is a FREE (- apart from exchange fees which are nominal for non-professionals) realtime, non-delayed tick datasource, including such excellent things as option chains and Level 2 data. We also intend to add the "Nifty" stocks from the Indian stock market. Advanced Edition is also undergoing a mild upgrade to version 1.2. Release should be around the end of May for both versions.

12 March 2007 Changes to daylight saving time in the US have caught us out!

10 March 2007 3000 users of Basic Edition; keep the feedback coming.

26 February 2007 Basic Edition 1.4 released; includes stocks for Indian and Australian stock exchanges; major new feature is the Option Strategy Browser.

1 January 2007 Spammers have started sending out emails using our domain; please ignore any unsolicited emails - they are not from us; we do not market ourselves in this way, and in particular we do not tout stock market tips.

15 November 2006 Preparing Basic Edition 1.4; we have had requests for support of Indian Stocks, so we shall we be offering Bombay Stock Exchange BSE and Nifty stocks in our next release, as well as updated stocks lists for all other groups, which we admit are a little out-of-date by now. 1.4 should be ready mid-December.

1 November 2006 Basic Edition 1.3 Minor bugfix release. We now have over 1300 users of our Basic Edition software - thank-you all, and please do not hesitate to offer feedback!

21 October 2006 Basic Edition 1.3 in preparation. Would the people who have been trying to download the advanced edition by pointing their browsers at http://www.stockwavesoftware.co.uk/download/install_stockwave_advanced_edition.exe and variations thereof, please note that the advanced edition is available by MAIL ORDER ONLY; it was hoped that online purchase via credit card would be available by now, but there appears to be a real problem with the "website guy" - it could be an extended holiday, a nervous breakdown, or a terminal case of asshole-itis ... no one knows - we seek him here, we seek him there ...

25 September 2006 Upgraded Basic Edition released - now at version 1.2; this is a collection of minor bugfixes and enhancements; some controls have been moved for better accessibility.

19 August 2006 New Contact page added, as the old one was a bit too complicated and er, ... rather crap.

18 August 2006 Installer glitches found in Basic Edition 1.1 - many apologies folks; now fixed, but if you are having any trouble at all, please email me direct at help@stockwavesoftware.co.uk

5 August 2006 Upgraded Basic Edition released - this is now version 1.1; new features are free streaming prices from the internet for UK, US and EU (- NB this is 15 minute delayed), and a 'Classical Technical Analysis' package for those of you who like that sort of thing. Windows 95/98/Me are no longer supported - the program should still work, but we are no longer specifically supporting these OSes.

21 July 2006 Minor visual alignment problems found on Basic Edition when using high screen resolutions (1280 by 1024 and above); fixed.

19 June 2006 Installer program can now handle non-standard Windows partitions - for those of you who have windows installed on your "O:" drive.

6 May 2006 Basic Edition installer re-instated; please contact if you have any problems with the installation process - please note that under Windows XP you may need to have Administrator-level privileges enabled.

30 April 2006 Added web-update for users of Basic Edition (- i.e. 'backfill' capability); this means you can now download FREE historical intra-day price data (15 minute resolution) for use in your analyses. This data will be updated weekly.

29 April 2006 Bug reported in Basic Edition installer; removed temporarily.

14 April 2006 Online, at last!

Latest News

Enquiries