Can't convert string to UTF-16LE for MD5 calculation on an Arduino

Question

TLDR

I need to convert text from a website into UTF-16LE format so that I can get the proper MD5 checksum but can't figure out how to go about doing that. This is all happening on an Arduino to log in to a router.

Background

I want to read values from a smart home device connected to a Fritz!Box router using an Arduino with an ethernet connection. The router has an open API and I'm using HTTP (EN|DE) to get the values. C++ programming is not my strong point.

I have been basing my attempts on BASH and javascript examples. For MD5 calculation I'm using tzikis' ArduinoMD5 library.

Steps

Based on the manual, you need to do the following:

Contact the router to get a challenge string (router sends XML)
Generate a response by calculating an MD5 checksum using the challenge string and password: <challenge>-<password>
Send the response, <challenge>-<response>, via POST
Receive the XLM answer from the router containing the SID
Use SID to request data or control a smart home device

I'm stuck on step 2. I have the challenge but cannot calculate the correct checksum.

The manual states:

The MD5 hash is generated from the byte sequence of the UTF-16LE coding of this string (without BOM and without terminating 0 bytes).

Attempts

So far I can request the challenge: 70067288 for example. The reponse appears to always be alphanumeric. My password is also alphanumeric.

unsigned char* hash2 = MD5::make_hash("1234567z-äbc");
char *md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print("MD5: ");
Serial.println(md5str2);

This was my attempt at confirmation. The manual gave 1234567z as a challenge example with äbc (the a in the example does indeed have umlauts) as a password. The response shown for this example was 1234567z-9e224a41eeefa284df7bb0f26c2913e2 but my checksum from the above code was 935fe44e659beb5a3bb7a4564fba0513.

I tried "manually" generating UTF-16LE and calculating the MD5 but that didn't work either.

byte rawTest[] = {0x3100, 0x3200, 0x3300, 0x3400, 0x3500, 0x3600, 0x3700, 0x7a00, 0x2d00, 0xe400, 0x6200, 0x6300};
char buffer[12] = {};
unsigned char* hash2 = MD5::make_hash(&buffer[0]);
char *md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print("MD5: ");
Serial.println(md5str2);

It gave me d41d8cd98f00b204e9800998ecf8427e.

I am not entirely sure the example given was correctly calculated. Using their example in the API documentation with BASH code from the site mentioned above, I get a different answer: 1234567z-bb6e5b7c7d4d485590f4e084ad3da989.

#!/bin/bash
# -----------
# definitions
# -----------
FBF="http://192.168.178.1/"
USER="root"
PASS="äbc"
AIN="AINOFYOURFRITZDECTDEVICE"
# ---------------
# fetch challenge
# ---------------
CHALLENGE="1234567z"
# -----
# login
# -----
MD5=$(echo -n ${CHALLENGE}"-"${PASS} | iconv -f ISO8859-1 -t UTF-16LE | md5sum -b | awk '{print substr($0,1,32)}')
RESPONSE="${CHALLENGE}-${MD5}"
echo $RESPONSE

Any help getting it working is greatly appreciatet.

Working Example (with caveats)

Requirements:

Password using only characters found in ASCII
Password at least 17 characters long

The password must use only ASCII characters due to the simple way it's being converted to UTF-16LE.

I can't explain why the password must be at least 17 characters long though. My code checks the length of the actual password programmatically.

With a password at least 17 characters long I get something like this:

Challenge: 6c607ee5

Challenge Pass (26): 6c607ee5-01234567890123456

Challenge Pass Preallocation Length: 26

MD5 (UTF-16LE): 3b26241ae4aab8eaf71d5c1599932178

Any password short and I get something like this:

Challenge: 621611e1

Challenge Pass (26): 621611e1-0123456789012345�

Challenge Pass Preallocation Length: 25

MD5 (UTF-16LE): 30a163ab8a267adfbb524b2b7412e81b

Please excuse any poor coding, C++ is not my strongpoint.

void fritzboxSID() {
  EthernetClient fritzBox;
  const char* pass = "01234567890123456";  // 17 ASCII character minimum
  const char* user = "fritz3456";  // Account on router, randomly generated or user created
  const size_t passLength = strlen(pass);  // Password length
  char c;
  uint8_t xml = 0;
  char challenge[9] = {0};  // Challenge
  char challengePass[8 + 1 + passLength];  // challenge-password
  uint8_t challengeCount = 0;
if (fritzBox.connect("192.168.200.1", 80)) {
    // Send login request
    fritzBox.println("GET /login_sid.lua HTTP/1.1");
    fritzBox.println("Host: 192.168.200.1");
    fritzBox.println("Connection: close");
    fritzBox.println();
while (fritzBox.connected()) {
  while (fritzBox.available()) {
    c = fritzBox.read();  // Read a single character from the router response
    if (c == '&gt;') {
      xml++;
    }
    if (xml == 5) {
        // Challenge starts after the fifth &gt;
        if (challengeCount &gt; 0 &amp; challengeCount &lt; 9) {
            // Save the challenge, character by character
            challenge[challengeCount-1] = c;
            challengePass[challengeCount-1] = c;
        }
        challengeCount++;
    }
    Serial.print(c);  // Print HTTP response
  }
}
challengePass[8] = '-';
for (size_t i = 0;i &lt; passLength; i++){
  // Copy password into the combined challenge-password string
  challengePass[i+9] = pass[i];
}

// Show challenge and pass, checking lengths
//      If the strlen(challengePass) doesn't equal the preallocation length then the MD5 will be incorrect
Serial.println(&quot;&quot;);
Serial.print(&quot;Challenge: &quot;);
Serial.println(challenge);
Serial.print(&quot;Challenge Pass (&quot;);
Serial.print(strlen(challengePass));
Serial.print(&quot;): &quot;);
Serial.println(challengePass);
Serial.print(&quot;Challenge Pass Preallocation Length: &quot;);
Serial.println(8 + 1 + passLength);

// Convert to UTF-16LE (only works for standard ASCII characters)
const size_t length = strlen(challengePass);
char buffer[2*length];
for (size_t i = 0; i &lt; length; i++) {
  buffer[2*i] = challengePass[i];
  buffer[2*i+1] = 0;
}

// Generate MD5
unsigned char* hash = MD5::make_hash(buffer, 2*length);
char *md5str = MD5::make_digest(hash, 16);
free(hash);
Serial.print(&quot;MD5 (UTF-16LE): &quot;);
Serial.println(md5str);

} else {
    Serial.println("Cannot connect to 192.168.200.1");
  }
}

If it helps, the response from the router for the initial login request is this:

HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: close
Content-Type: text/xml
Date: Fri, 16 Sep 2022 08:21:27 GMT
Expires: -1
Pragma: no-cache
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'none'; connect-src 'self'; font-src 'self'; frame-src https://service.avm.de https://help.avm.de https://www.avm.de https://avm.de https://assets.avm.de https://clickonce.avm.de http://clickonce.avm.de http://download.avm.de https://download.avm.de 'self'; img-src 'self' https://tv.avm.de https://help.avm.de/images/ http://help.avm.de/images/ data:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; frame-ancestors 'self'; media-src 'self'
<?xml version="1.0" encoding="utf-8"?><SessionInfo><SID>0000000000000000</SID><Challenge>621611e1</Challenge><BlockTime>0</BlockTime><Rights></Rights><Users><User last="1">fritz3456</User><User>UserName</User></Users></SessionInfo>

Good find. My password is ASCII only. The example provided in the documentation was not. How would you convert it to UTF-16LE in Arduino C++ in that case? I still need to programmatically convert the challenge as well. Thanks for the quick answer. — Jeremy, Sep 15 '22 at 07:16
byte rawTest[] = {0x3100, 0x3200,..... this is not doing what you think it does. I'm assuming you weren't looking for a creative way to fill your array with 0x00 values. Probably you meant 0x31, 0x00, 0x32, 0x00,..... — timemage, Sep 15 '22 at 15:53
Good to know. If I ever get around to expanding the code from ASCII passwords I'll try that out. Let's see how hard it is to complete the login process and get data first. — Jeremy, Sep 16 '22 at 08:35

Edgar Bonet · Accepted Answer · 2022-09-16T20:21:43.490

You have to convert the string "<challenge>-<password>" to UTF-16LE. If the whole string is plain ASCII, the conversion is trivial: you just have to add a zero byte after each ASCII byte.

// ASCII string to be hashed.
const char* text = "1234567z-abc";  // umlaut removed
// Convert to UTF-16LE.
const size_t length = strlen(text);
char buffer[2*length];
for (size_t i = 0; i < length; i++) {
    buffer[2*i] = text[i];
    buffer[2*i+1] = 0;
}
// Compute and print the hash.
unsigned char* hash2 = MD5::make_hash(buffer, 2length);
char md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print("MD5: ");
Serial.println(md5str2);

Note that I use the two-parameter version of MD5::make_hash() in order to pass the length of the buffer. The one parameter version of this method would stop at the first zero byte.

Edit: Regarding your extended question (the new section “Working Example”), this is a problem completely unrelated to the previous one. It is about string termination.

In C and C++, strings are arrays of characters terminated with a NUL ASCII character (numeric value zero). The terminating NUL is not part of the string per se, but it should be stored in the array holding the string. String literals are interpreted by the compiler as NUL-terminated arrays of characters. Every function that expects a string as an input expects it to be NUL-terminated.

In this particular instance, you failed to NUL-terminate challengePass. When this array is given to strelen(), this function scans the memory until it finds a zero byte. If that zero byte happens to be right after the end of the array, your code will work as expected. Otherwise, any non-zero byte coming after challengePass in memory will be interpreted as being part of the string. The program behavior is unpredictable, because you do not know what will be stored in memory right after challengePass.

The solution is to NUL-terminate the string. Start by allocating one more byte:

char challengePass[8 + 1 + passLength + 1];  // challenge-password

Then, right after copying the password, add a zero byte to terminate the string:

challengePass[8 + 1 + passLength] = '\0';  // terminate the string

Note that '\0' means “the character with numeric value zero”.

Alternatively, you can terminate the string right after adding the dash, then use strcat() to concatenate the password. strcat() will take care of terminating the string:

challengePass[8] = '-';
challengePass[9] = '\0';  // temporarily terminate the string
strcat(challengePass, pass);  // append password

That actually works, but for some reason I generate text it fails. If I then manually type the new challenge and same password into const char* text... it gives the correct MD5.
C++ character/string operations are profoundly confusing. — Jeremy, Sep 15 '22 at 12:36
@Jeremy: What do you mean by “_I generate text it fails_”? In what manner does it fail? I should not matter whether the text is typed in the source or generated programmatically. — Edgar Bonet, Sep 15 '22 at 13:20
I tried posting code in a comment but it was a complete mess.
const char* pass = "myPassword"; const size_t passLength = strlen(pass); char challenge[9] = {0}; // Challenge char challengePass[8 + 1 + passLength]; // challenge-password

Using the ethernet module I then read in the XML from the router, copying it character by character to challenge and challengePass. After that I tried strcat to build the challenge-password string to feed into the MD5 function. The MD5 was always wrong though. Is it because strcat adds a null-terminator to the end? — Jeremy, Sep 15 '22 at 14:00
@Jeremy: The question about string concatenation in your last comment is: 1. Unrelated to the current question, which is about ASCII → UTF16-LE conversion. 2. Impossible to answer without seeing how you are actually trying to concatenate the strings. — Edgar Bonet, Sep 15 '22 at 16:01
This entire question is in the context of assembling a string to not just generate any MD5 checksum but the correct MD5 checksum. The particulars are very important. I understand this is Stack Exchange and tangential questions should be asked on their own. Now to see if I can figure out the other steps required. Thanks again for the help. If I, or someone else, figures out the password length caveat I'll update the solution. — Jeremy, Sep 16 '22 at 08:31
I don't know if it's worth changing anything in the answer for this, but it may be good to know that you seemingly have the option of using a lower level set of features (MD5_CTX, MD5Init, MD5Update, MD5Final) that provide for avoiding the temporary buffer, dynamic memory allocation, and concatenation. The various concatenations, including those of 0x00 being done to up convert ASCII to UTF-16LE, become separate calls to MD5Update. — timemage, Sep 16 '22 at 22:53
Thanks for the tip. I'm slowly in the process of converting it all to useable functions. Once I get to optimization I'll give it a shot. No doubt there are plenty of ways to optimize my C++ code. — Jeremy, Oct 06 '22 at 10:55