[I know, I know. You're probably thinking: Oh great, the start of another unfinished series by Rafael. Sorry. The RemoteFX series indeed suffered a setback. You will see the conclusion of this story, promise.]
Getting your feet wet
A month or so ago, Jenny pinged me about a toy her sister had collected from a yard sale. The toy? A USB-based Hello Kitty figurine that made noise when you typed. She had plugged it into her Windows 7 PC and encountered a slight problem: It didn’t do anything. This thing needed software. After an Internet deep dive, I discovered and downloaded the drivers on Cables Unlimited cached site at about 1 kilobyte/second. I carefully extracted the ZIP and ran Setup...
I’m remember experiencing a number of emotions at this point, but the only one that made an impression (on my desk) was rage. Someone had packaged the software using an old school version of InstallShield (3.0) that produces 16-bit stubs. After calming down, I decided to go ahead and rip out the binaries from its cabinet files using an equally as old tool called WinPack.
With files in hand, I was surprised at what little there was to play with. There was a documentation attempt, an icon, a little notification area program, a sound file, and a supporting DLL. I plugged the toy into the PC and ran the program. Other than putting a cutesy icon in my notification area, nothing happened. Sweet. Time to debug!
I first threw the executable into IDA Pro. It was immediately obvious it only handled UI operations. The actual work was delegated to TenxHID.dll. I then started analyzing the library. TenxHID.dll contained three exported functions (four if you count DllEntryPoint):
Focusing first on DeviceIni, I followed the various calls around and drew a mental picture of how this thing worked. It first looked for the HID device using HidD_GetHidGuid, SetupDiGetClassDevsA, SetupDiEnumDeviceInterfaces and SetupDiGetDeviceInterfaceDetailA APIs. Then, using WriteFile, it communicated with the device over a pipe. With my debugger, I set a breakpoint on various calls to determine a] if the library was finding the device and b] if WriteFile succeeded.
Surprisingly, the library was finding the device just fine. No problems there. But WriteFile was failing, due to an invalid HANDLE (hFile) being passed in. Tracing my way backwards, I found and set a breakpoint on the related CreateFile. A re-run through the debugger confirmed CreateFile was failing with an ERROR_ACCESS_DENIED.
Whoa, hold on there
Before I continued, I decided to take a few steps back. I had broken one of my rules when dissecting software – observe the software in an operating manner first. I set this rule because more often than not, the very software I’m tinkering with doesn’t work in the intended environment, let alone mine.
I fired up VMWare and installed Windows XP. After fighting VMWare’s broken Windows 7 SP1/USB support (a story for another day), I was up and running. (Any virtualization software would have worked here, although I haven’t tested Hyper-V w/ RemoteFX and USB redirection yet.) I plugged in the device, installed the software and the device was spinning around making lots of noise. Okay, cool.
Back to digging
I shifted my focus back to why this call was failing. The call looked similar to:
CreateFileA(lpFileName, GENERIC_READ | GENERIC_WRITE,FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, 0, OPEN_EXISTING, 0, 0);
Although the inputs worked on Windows XP, I decided to start playing with the parameters. I set a breakpoint on CreateFile and started altering values before the call. Removing the GENERIC_READ did the trick, but why?! I couldn’t find any documentation on this but I suspect Microsoft made a tweak to the way Windows handles communication to and from HID devices. Using USBView, I verified the toy exposed two USB endpoints that were both receive-only.
Endpoint Descriptor: bEndpointAddress: 0x81 IN Transfer Type: Interrupt wMaxPacketSize: 0x0008 (8) bInterval: 0x0A Endpoint Descriptor: bEndpointAddress: 0x82 IN Transfer Type: Interrupt wMaxPacketSize: 0x0008 (8) bInterval: 0x0A
So it makes sense that Windows would prevent READ operations to occur here; there isn’t an OUT endpoint available to accommodate a READ request. But I’m not sure why Microsoft wouldn’t just let ReadFile fail – the very behavior the library was looking to handle. (Insight into this change would be appreciated.)
Anyway, having confirmed the toy has no outputs I simply altered the library’s assembly (with a hex editor) and changed the flags from 0xC0000000 (GENERIC_READ | GENERIC_WRITE) to 0x40000000 (GENERIC_WRITE). I re-loaded the executable and the device spun up and made its introductory noise!
But wait, I pressed a few keys on my keyboard and nothing happened… Dum dum dummmm! (To be continued.)