TradeKing's Site Outage

Everyone: As many of you know, we experienced a broad site outage today that impacted our phone support as well. Those issues began when our clearing firm upped its margin requirements at midnight last night for naked options positions. This move in extremely volatile markets effectively increases the value of “collateral” against trades with unlimited risk and shields both TradeKing and our clients from potentially ruinous losses. Upping margin requirements overnight is an extraordinary move but was one necessitated by our clearing firm. Our development team stayed up all night adjusting our systems to this change, but that margin change compounded with historically unprecedented market volume cascaded into broader site issues. Phone volume was extremely high, too, with some clients getting a busy signal as a result.

Many of you have speculated as to what went wrong, with theories ranging from site hacking to insufficient server capacity. In fact, the root problem was neither of these. We upgraded our capacity after previous outages as promised, and those upgrades delivered in a pinch. In the last few market days, we’ve handled spectacularly high trade volumes with only isolated slowdowns. Obviously, today’s site performance doesn’t fit that description.

For those who’ve asked for specifics, here’s what happened and how we fixed it:

Late last night, TradeKing's clearing firm, Penson Financial, increased margin requirements on option positions by 100%.  These requirements are sent to TradeKing via file every night, along with multiple other files.  Normally, TradeKing receives these files in the early hours of the morning and has plenty of time to import the data into our databases. 

However, due to the change in the margin requirements and/or the sheer volume of trading and money movement activity from the previous day, TradeKing did not receive the balance file, which is the source of start-of-day balance numbers, until after trading had already started this morning.  This was an extremely rare occurrence. 

To ensure normal trading would commence without an issue, a system that contains the previous trading day's unofficial numbers was used as the backup to generate balance info and to compensate for the effect of the late arrival of the balance file from Penson.  This system was never intended to be used during a normal trading day, it's purely an after-hours system.  Unfortunately, this system could not handle the amount of requests that was being asked for by the website.  The root cause was this alternative system, and the fact that we deployed it in a way that we had never tested.  It was crunch time, and we made the call to try it in the hopes that it would enable trading for everyone in that pinch. It didn't work.

The issue that made it fail under stress was tracked down and corrected, which then returned trading systems to normal.  Again, this alternative system was not intended to replace the balance process typically in place on TradeKing, but due to the late arrival of the daily balance file, this system was used as a backup to get each of you balance numbers as close to accurate as possible.
 

Now let me speak to a broader question many of you have (justifiably) asked. Namely:  why should I continue to trust TradeKing with my money in light of an experience like this? It’s a fair question. Here are a few thoughts:

All market players, including many of our competitors, experienced slowness, intermittent outages, busy signals, etc. during this massive market volatility. As much as all online brokers strive for 100% uptime, historic volatility and trade volumes sometimes exceed peak capacities or make systems behave in ways that cannot be anticipated or tested in advance. It’s a painful fact of our business.

You may be thinking:  I didn’t choose TradeKing to be AS GOOD as the other guys; I chose you to be BETTER. Again, fair point. Given the crisis-level conditions in markets globally in recent days, on balance we’ve performed as well or better than many of our peers – but not optimally, and certainly not to our ideal standards. 

Here’s how we differ from the other guys:  Not all brokers are prepared to face their clients and admit that things weren’t perfect. Many brokers don’t staff up their phone teams, work from midnight onwards (and beyond), or distribute free-trade credits as freely as we’ve done. Very few offer public forums where clients can freely share information, vent their frustration, or interact with senior management. And practically none of them can say busy signals are rare for their clients, which is true for us.

That said, we don’t intend to make a habit out of outages, or busy signals, or public apologies like these. Sincere apologies do matter in these situations, and I would never skip over that part. Standing shoulder-to-shoulder with the TK team, let me say:  we’re very sorry to have frustrated or inconvenienced you this way. That said, apologies matter less than simply putting your head down, doing right by clients, and logging more outage-free days from now onward.

TRADE FOR FREE THIS FRIDAY

This market has been rough on all of us, and we know you deserve better. To do right by you, we’re announcing that Friday, 8/12, will be a commission-free trading day for all TradeKing clients. Watch your inbox and this space for more details soon.

If you continue to experience any account inaccuracies, troubles logging in – anything at all – please email us at service@tradeking.com or call 877-495-5464.

We welcome your questions or any follow-ups you may have.

Be Good,

Don Montanaro

Chairman & CEO, TradeKing