RSA SHA256 signing in iOS and verification on Java

Question

1. I generated an RSA key pair with SecKeyGeneratePair. The key size in bits is 2048.

   NSDictionary *privateAttributes = @{(NSString *)kSecAttrIsPermanent: @YES, (NSString *)kSecAttrApplicationTag: PrivTag};
   NSDictionary *publicAttributes = @{(NSString *)kSecAttrIsPermanent: @YES, (NSString *)kSecAttrApplicationTag: PubTag};
   
   NSDictionary *pairAttributes = @{(NSString *)kSecAttrKeyType: (NSString *)kSecAttrKeyTypeRSA, (NSString *)kSecAttrKeySizeInBits: @2048, (NSString *)kSecPublicKeyAttrs: publicAttributes, (NSString *)kSecPrivateKeyAttrs: privateAttributes};
   
   SecKeyRef publicKeyRef;
   SecKeyRef privateKeyRef;
   OSStatus osStatus = SecKeyGeneratePair((CFDictionaryRef)pairAttributes, &publicKeyRef, &privateKeyRef);
   switch (osStatus) {
   case noErr:
       break;
   default:
       break;
   }
   

2. Create the X.509 format of the public key and send it to the server.

3. Create the SHA256 digest of the custom string with CC_SHA256.

   NSMutableData *hash = [NSMutableData dataWithLength:(NSUInteger)CC_SHA256_DIGEST_LENGTH];
   NSData *data = [stringToSign dataUsingEncoding:NSUTF8StringEncoding];
   CC_SHA256(data.bytes, (CC_LONG)data.length, hash.mutableBytes);
   

4. Sign the string with SecKeyRawSign method using kSecPaddingPKCS1SHA256.

   // Sign the hash with the private key
   size_t blockSize = SecKeyGetBlockSize(privateKeyRef);
   
   NSUInteger hashDataLength = hash.length;
   const unsigned char *hashData = (const unsigned char *)hash.bytes;
   
   NSMutableData *result = [NSMutableData dataWithLength:blockSize];
   
   uint8_t *signedHashBytes = malloc(blockSize * sizeof(uint8_t));
   memset((void *) signedHashBytes, 0x0, blockSize);
   size_t encryptedDataLength = blockSize;
   
   OSStatus status = SecKeyRawSign(privateKeyRef, kSecPaddingPKCS1SHA256, hashData, hashDataLength, signedHashBytes, &encryptedDataLength);
   
   NSData *signedHash = [NSData dataWithBytes:(const void *) signedHashBytes length:(NSUInteger) encryptedDataLength];
   

5. Apply base64 on the signed data and send it to the server.

6. The java server cannot verify it with the public key.

I have the same above code in Swift. As a debug step, I've exported my private key too and tried to follow the exact same steps in java. Until step 3 everything is the same. So, the iOS creates the same digest as the java app. The fourth step, the signing creates a different output than the java code.

Here's the java code:

final Signature instance = Signature.getInstance("SHA256withRSA");
instance.initSign(privateKey);
instance.update(MessageDigest.getInstance("SHA-256").digest(rawString.toString().getBytes("UTF-8")));

Answer 1

Digital signature API for iOS and Java is different but the result is the same.

iOS SecKeyRawSign with kSecPaddingPKCS1SHA256 uses a SHA256 digest, but in Java Signature.sign requires the raw data and it makes digest+pkcs1. Use

instance.update(rawString.toString().getBytes("UTF-8"));