The privacy of thousands of people has been seriously compromised for an unknown period of time -- probably many months -- by the online computer sales operation known as ComputerHQ.com.
When I discovered this company's gross mismanagement of their customers' personal data, I gave them every benefit of the doubt as I sought to help them correct it. I never believed I would be assaulted with lies and misinformation, accused of hacking and other illegal acts, essentially made a scapegoat by the company's irresponsible principals.
This is the true story of my discovery; everything that led up to it, and everything that followed. Every detail is here.
Here I explain in full, the ridiculously simple "exploit" that exposed every record in the company's order database, including full personal details of every customer, every purchase, every credit card number. It exposed all this to anyone, anywhere with a Web browser. It was in fact not an exploit at all, but an almost unbelievable disregard on the part of ComputerHQ for the simplest attention to security safeguards to protect their customers' personal and private information.
I will show how ComputerHQ bungled the correction of this inexcusable exposure not just once, but three times, beginning with my first alert to them on Saturday, the 16th of June; needlessly and stupidly extending the exposure of these sensitive records for the greater part of two days. Their grossly insecure server was online long enough for every single record in their database to have been stolen by any number of people, anywhere, many times over.
They have yet to take proper security measures. Nor to my knowledge, have they made any effort to inform the thousands of affected people of their exposure to potential intrusion and loss. Quite the contrary. They have railed against a reporter who contacted some of those exposed clients. They have maligned me falsely, referring to my discovery of their stupidity as "hacking." They have accused me of illegal acts I have not done, and characterized my inadvertent discovery as a deliberate hunt for vulnerabilities -- which, they claim without any basis, I regularly perform in order to gain publicity. Furthermore, they have intentionally mischaracterized the nature of the problem and the means by which I gained access.
Nothing could be further from the truth than their fictions. I mean to set the story straight.
At this writing (19 June), the story has been covered in part by Michelle Delio of Wired and by Mark Hachman of ZDNet's ExtremeTech. But there's plenty more to tell, which those writers didn't know or had no time or space to include. And the unfortunate, grossly inappropriate response of ComputerHQ's principals came after those articles were published.
This is a long tale, because I seek to make it as complete and well-documented as I possibly can. Please bear with me.
On the 10th of January 2001, a longtime client paid me a visit. She wished to buy a new computer. We had arranged to meet and make an online purchase, and have the new computer system delivered to my shop, where I would then set it up for her purposes.
After some checking on Pricewatch.com and some other sites, we decided that of those sites we'd seen, ComputerHQ.com offered the most suitable items at reasonable prices. Their build-your-own website mechanism suited my desire to choose components carefully, and their choice of items seemed just right.
We made our choices, and placed the order with her credit card.
On the 2nd of February, 23 days later, the system was delivered. It had taken too long, but it was as ordered and the system had no significant problems. My client was very pleased with her computer.
Some weeks ago, that system's hard drive suffered a serious failure; lost sectors and corrupted data rendered the system non-functional.
I contacted ComputerHQ and arranged for a replacement drive. With my own credit card as security, they shipped the new drive immediately, while I agreed to send them the old drive once the new one had arrived. They would charge my card only if they did not receive the faulty drive.
Their usual procedure is to take shipment of the bad part prior to shipping out a new one; which may be why they apparently kept poor records of this different but not-so-unusual arrangement.
So it was that when they finally received the faulty drive, the recipient found no record of the prior shipment, and assumed it required a replacement; and so they shipped out another new drive.
I received that package on the 13th of June. I was puzzled, but too busy to deal with it until the following Saturday.
Here you'll find a captured record from the UPS website which documents the shipments I received from ComputerHQ, except the first replacement drive for which I lack a tracking number.
So it was that last Saturday afternoon, I got around to examining that package and its accompanying invoice.
I located the first invoice from the original computer purchase. I compared it with the one that had just arrived. Their content was substantially the same. There was nothing to indicate why I'd received the hard drive.
Back in February, I had noticed a URL printed across the bottom of the first invoice. (It's not a common feature of invoices.) Now, seeing it again, it caught my eye yet again.
This time, I did what any ordinary propellerhead would do. I typed it in on my browser.
I was nothing less than stunned when the same record I had in my hand was now displayed, in full, on my monitor.
I hit the Reload button just to make sure. There it was. My client's name and address. My address. Her credit card number, complete with expiration date. The whole order. Every detail.
I understood what I was looking at. Wouldn't anyone? Their web server was really and truly handing out their records.
The record I had accessed was numbered 8420. That number was in the URL, and it was blazoned upon the printed page. I tried changing the number to 8419. Another record came up, also complete to the last detail, credit card and all.
I input several more numbers in sequence. Every one of them produced another detailed record, another name, another credit card number. They all looked like this one -- which is very real incidentally, I've left it looking almost exactly as received; but it's a dummy record which contains no valid data.
Realizing the magnitude of this security hole, I purposely retained a number of the records, for my own protection and as proof of what I was observing. In order to gain some understanding of how many people and records were compromised, I tried higher numbers until I found the highest: number 16453. Then I tried lower numbers until I had zeroed in on the first record: number 1301, dated 21 July 2000. I saved those two with the others.
It was about 2:45 PM. Using ComputerHQ's toll-free phone number, found on my invoice, I called ComputerHQ. I used the toll-free number intentionally. It would create a permanent record of the call. I would hide nothing.
A woman answered. I asked for their computer system administrator. She passed me to someone with a faint Asian accent who identified himself only as John.
I realized John could identify me by my association with the shipping error they'd made. So I worked out the details of that order first. Yes, it was an error. John wanted me to ship the drive back, to which I agreed.
Then I dropped the bombshell. "You have a very serious problem with the security of your customers' data." "I have full access to every order in your database, credit card numbers and all."
John replied that that wasn't true. That I would need the ZIP Code for each before I could see the orders. No, I said, I don't.
I gave John the URL I was using. I explained that it was printed on my invoices. I pointed out that it was presumably printed on every invoice they shipped. I recommended that he take their system offline immediately and leave it offline until this huge security hole was fixed. I asked if he'd be willing to inform me of the result, and mentioned my concern that those whose personal data had been exposed should be informed. He agreed on all counts, and our call ended.
Minutes later, as I monitored it, the website went offline.
My first concern was for all those cardholders, whom I estimated to be around 10,000 in number. I decided I should try to report this huge breach to someone in the credit industry.
Lacking anywhere else to start, I turned over my own MasterCard and dialed the number I found there.
That was an interesting exercise. The woman I reached had absolutely no clue how to respond to my disclosure. She painstakingly explained to me, several times, that the cardholder accounts are the responsibility of some 22,000 banks throughout the world. She actually told me I had to contact those banks! All of them!
No amount of explanation would do. This woman was a seriously slow-witted individual. I asked for her supervisor.
There I found an intelligent human. Although such incidents were not part of his job, he knew whom to contact. I gave him all the details I had, including the merchant account number for "LJ Systems" which appeared on the credit card receipt that had been stapled to the original order invoice.
That done, I reflected. Realizing this incident would be of interest to certain others, especially to people who are concerned about privacy issues, I posted this message to a privacy-oriented newsgroup, grc.privacy. (That link to my message leads to a large textfile. That text is the totality of a two-day discussion in which many people took part.)
In that newsgroup message I disclosed absolutely nothing exploitable. Even though I had every reason to believe the problem would be dealt with and would not reappear, I also knew that mistakes could happen. I knew that if there was one such vulnerability on the ComputerHQ.com site, there might easily be another. If it became known who they were, others might go there seeking illicit entry. Further, though I firmly believe in full disclosure, I saw no purpose at that point in exposing ComputerHQ to public scrutiny; they deserved a chance, I reasoned, to deal with the situation first.
For all those reasons then, I pointedly did not name the site, and I gave no useful details whatsoever about the nature of the "hole". (Hole? It'll have to do. I know no word for "hole so big it can't be described as a mere hole".)
Much lively discussion ensued, in which I participated at some length.
It was suggested that I should disclose the name of the company. I declined, and explained why.
I was questioned from a variety of angles. I responded with the facts, though always without disclosing any "exploitable" facts.
It is a very, very good thing I didn't release the details.
It was around noon on Sunday before I took the time to check the ComputerHQ site again.
Inexplicably, unbelievably, the site was back up and there was absolutely no change in the situation. The records were still, in effect, being offered like tempting little candies to all the world.
It was then that I retrieved from ComputerHQ, record number 16454, which had not existed the previous day. The reader will note that it is time-stamped 3:11:20 AM. I fetched record 16455. It showed that at 4:16:17 AM PDT, someone in Fairport, NY had placed an order.
The site had been open for business, and I do mean open, since about 3:00 AM!
I tried to phone ComputerHQ. Their phone rang endlessly. No one home, and not even an answering machine.
OK, sure, it was Sunday. They were closed. But the whole world wasn't closed, and most of all, their website, which was accessible to the whole world, most certainly was not closed! What was I to do?
This situation now had me really worried. In a vacuum of information, I found myself sucking in all sorts of explanations for this mind-boggling circumstance. Did a hacker have control? A crooked employee? Were they idiots, criminals, neither, both?
I posted a brief message to grc.privacy, making known this mind-bending fact.
I decided I needed witnesses. I contacted some very trusted friends by email, provided them the "evil" URL, and asked them to just quietly take note of the situation and tell me their thoughts. They did so.
In the process, we realized that the website script (print.asp) seemed to be relying on a Javascript popup to hide the record it was sending. The popup alert (which I hadn't seen till then; I never leave Javascript enabled on my browser) demands a ZIP Code, and appears, at a glance, to deny access to the record. While the popup is displayed, the record is not visible.
A javascript popup bears no resemblance whatsoever to a security measure. But it might explain why a rank amateur at ComputerHQ could conceivably believe the script wasn't handing out readable records. Not that this was very informative. It was already a foregone conclusion that someone over there wasn't too clever.
And the problem remained. The site -- and all those records -- were accessible right then. Something needed to be done immediately to get those records offline.
Someplace, probably at home, were one or more owners or managers of ComputerHQ. My task was to find such a person and reach him or her.
I retrieved the domain record for COMPUTERHQ.COM. This produced a name and an address: Ted Chan, 26046 Eden Landing Rd., Hayward, CA.
The phone numbers were the same ones no one was answering. Unfortunately, domain records can be useless in determining who runs a business. The names on those records may belong to a website manager, a long-departed employee, a staff member at the domain hosting service -- one never knows. I needed more.
The domain record showed Ted's email address: tedc@LJSYSTEMS.COM. This was hopeful. LJ Systems was the name on my client's credit card receipt. This could be the "parent" organization. So I retrieved the domain record for LJSYSTEMS.COM.
Not much use. I had another name and address: Joe Kim, 26203 Production Ave. #7, Hayward, CA.
What I needed now was corroboration of some sort. Another source so I knew I really had the name of a principal of the company.
I surfed to the website of the California Secretary of State. I located the Business Search page, which thankfully does allow name searches which produce partial corporation records.
After a few false starts (there's no LJ Systems Corporation in California and no active corporation with a remotely similar name), I found this record for ComputerHQ.Com, Inc. Aha! There's Ted Chen again. A different address, but that's fine. Ted is surely directly involved with ComputerHQ.
I wended my way to Google. I typed in Ted's address from the corporation record: "1807 Fumia"
Bingo! A stroke of sheer luck. Ted's house is for sale! It's on a real estate agent's website. Nice place, its asking price is $768,000.
Now, something I know about real estate agents: They answer their phones. Especially when they have $3/4 million houses on the market. They also have excellent contact information for their clients. Those hefty commissions are a fine incentive.
The agent answered her phone (no surprise). Knowing Ted might not own that house anymore, I decided to play it slick. "Hi, I'm calling about Ted Chen's house, at 1807 Fumia Place. That's the one, isn't it?"
It was. She'd confirmed he was her client. I explained the situation. Thousands of people's credit cards were being compromised on Ted's website. I had to reach him. Despite herself (she seemed dubious), she agreed to contact him immediately and pass on my name and number.
That was about 1:30. Three hours later, at about 4:20, Ted called. Ted listened. He had considered the problem solved (I have no idea why). I explained in detail, for the second time. The URL, I told him, which was printed on their invoices, produced the data for anyone who wanted it. He gave me his email address. I sent him the URL by email.
An hour or so later, ComputerHQ was offline again.
As ever, the discussion of the incident continued on grc.privacy. I was asked again to disclose the business name. Again I declined, this time not primarily because of the security concern. I was totally confident the particular exposure I'd found would be remedied.
Rather, I considered the company should now have an opportunity to deal with the problem themselves. I preferred, for the time being at least, to extend them my trust, and the courtesy that trust would imply.
One of the persons to whom I had given the URL on Sunday was a journalist; not just any journalist, but an individual I know and trust implicitly. Accordingly, I sent that person a copy of my moderating message, in which I argued against immediate exposure of the company. I knew he'd read it and take heed.
On Monday morning just past 7:00 AM, I got an email from Michelle Delio of Wired. One of the friends to whom I'd given the URL, who wisely took a far less forgiving view than I of ComputerHQ's irresponsibility, had passed on the URL to Michelle.
Hearing from Michelle was no great surprise. But what she told me was a surprise. The hole was still open. She could view the ComputerHQ records. Could I confirm?
Confirm I did. The site had been up, and wide, wide open, since at least 6:30 AM.
At some point, we all run out of patience. I think it was at about this point that I had reached my limit.
I told Michelle that I had no further concern about withholding the company's name. Once again though, I could not do so until the breach was closed and I pointed out, neither could she. For the moment, their own incredible, inexcusable, undeniable and complete stupidity and irresponsibility was oh, so very ironically shielding ComputerHQ from public exposure.
I posted another unhappy bit of news to grc.privacy. I said, and I meant it, "I have just lost all confidence in them."
I completed an email interview with Michelle. I was also contacted by Mark Hachman of ZDNet's new ExtremeTech.com (an associate of the aforementioned journalist friend), and did a phone interview with Mark. Both Mark and Michelle had now indicated their intention to name ComputerHQ.com in their articles.
Around 8:45, I received an email from Ted Chan. He indicated that the script was now repaired. Indeed, the URL no longer produced any confidential data, only the bit of popop script. On a browser without Javascript, it's a blank page.
It had certainly taken them a while.
The Wired article was online by about 4:00 PM. Michelle had contacted a number of the people whose records she'd found on ComputerHQ. Their comments were scathing. Those people were, very understandably, upset!
And well they should be.
Even so, I don't doubt that ComputerHQ was overwhelmed. If not from irate phone calls (I heard they weren't answering), then by their own recognition of the sheer magnitude of this disaster. I have no doubt that the principals of ComputerHQ/LJ Systems (and whatever other names they use) were beside themselves. The fallout from their irresponsibility could potentially ruin their businesses utterly. The fact that it had become a media event must have magnified their shock.
The ExtremeTech article appeared online sometime Monday evening. Mark had done the same as Michelle, interviewed some of the exposed ComputerHQ customers. It wasn't pretty.
All through this incident, I had given ComputerHQ every benefit of the doubt. I was not complimentary. Their irresponsibility was self-evident. But at no point had I done anything but attempt to help them. My focus had been on the victims, those thousands of people.
All along I knew, but didn't really believe, that ComputerHQ might choose to try and sweep this under the rug. Or worse, to accuse me of wrongdoing. It was with thoughts like those far in the back of my mind that I chose to place the facts, as possible, before the public in grc.privacy; to contact MasterCard; and to quietly inform trusted friends, including one in the media.
Now I am very, very glad I did all those things.
At 6:00 PM Monday, Michelle Delio forwarded me this text -- an email from someone calling himself only Joe. His email address, joe@ljsystems.com, indicates he's for real. There exists no LJ Systems Corporation in California, itself a curious fact. But LJ Systems, whatever it is, is unquestionably involved with ComputerHQ; so Joe really is quite evidently a principal or spokesperson for ComputerHQ.com.
Joe's email is a mass of obfuscation and falsehood. I've answered it here.
At 8:30 PM, Joe sent me another email, with copies to Michelle Delio, the Wired reporter, and to a couple of unnamed people at ljsystems.com. In that message, he yet again as much as accuses me and Michelle of illegal acts. I've answered that message here.
Several hours later, an individual who had emailed Joe in protest of his exposure, got yet another email from Joe, apparently biolerplate that LJSystems had authored specifically for the purpose of responding to irate customers' emails.
In that email, Joe intensified and expanded upon his attack on me and on the journalists who'd so soiled the noble name of ComputerHQ. I've answered that one as well.
At the time I discovered the exposed data, and until the shock and disappointment of Monday's events, I gave little thought to the overall security of the ComputerHQ server.
What I had found was, after all, a scripting error and a serious mistake with respect to access permissions. It was not related in any way to security vulnerabilities of their server, nor any of the (potentially many) expolits thereof.
Fortunately, I frequently use the Network Tracer at the slightest excuse. I use it to help identify entities on the Net, to follow up on firewall alerts, the origins of emails, and so forth.
The Tracer retains the records it retrives. So it is that I can see a little of the recent history of the ComputerHQ.com domain name, including responses from their web server to the queries of the Tracer.
In addition, starting Monday evening, I became personally interested in just what is the state of affairs at ComputerHQ/LJSystems with respect to security. I began actually investigating; whereas, contrary to Joe's assertions I had previously done no such thing.
I found Joe's statements both revealing and evasive, and his failure to clearly identify himself also piqued my interest. I began to examine ComputerHQ.com and the associated LJSystems entity more closely.
This portion of the tale will become a bit technical. If the technicalities confuse you, pay attention instead to the conclusions I draw. That's what's important to the story.
I have analyzed some of the available data about the security of ComputerHQ.com. In retrospect, I see that I should have paid closer attention to my own trace data when I was in the process of choosing a vendor.
Combining past and recent observations casts some useful light into the murk.
Joe's claim that ComputerHQ.Com was recently moved is not well supported. A prior business name for the enterprise, is revealed. The close connection between ComputerHQ.com and the enigmatic LJ Systems Corporation is further documented.
The probable state of their server's security, from January up to the recent past, looks very poor.
More to come...