Implementing Hill Cipher for data encryption

2025-11-16

Implementing Hill Cipher in C++

Everyone knows that encryption/decryption is very important when writing malware. That can be used for everything, starting from obfuscating strings and shellcodes and ending with encrypting traffic between malware and the C2 server in order to avoid detection.

What is Hill Cipher?

The Hill cipher is a very interesting algorithm that is based on linear algebra and uses matrices to encrypt/decrypt words. It represents characters with numbers and uses those numbers to do calculations with matrices. Let me explain it more easily. Let’s say there is a string that needs to be obfuscated: VirtualProtect. This is the name of a function that is part of WinAPI that changes protection on allocated memory. The thing is that AVs often check if there is VirtualProtect function call after VirtualAlloc. Mostly, when malware developers use those functions, they do this:

  1. They allocate enough memory for a shellcode with VirtualAlloc and set permissions PAGE_READWRITE. This permission tells the CPU that it can write/read data in that memory region, BUT it cannot execute instructions from that memory region.
  2. The attacker writes malicious shellcode into that memory region.
  3. The attacker calls VirtualProtect to change the permission of that memory region, which often is PAGE_EXECUTE_READWRITE. Finally, malware creates a new thread that executes written shellcode.

This behavior is often flagged as malicious (because mostly, it is!) by AV engines, and the process gets killed and the malware deleted.

I already talked about function name obfuscation and loading functions dynamically here. That is a very good recommendation, but it has one problem: those strings are unencrypted, and even static analysis can detect them. That’s where the Hill Cipher comes in. It can be used for hiding those function names.

The Hill Cipher works this way: it takes a matrix as a key and a vector as data. It does multiplication on them, and the output vector is encrypted data. If we want to decrypt already encrypted data, we will have to have the inverted matrix of the key matrix in order to get the real value. The process is the same.

Implementing the key

First of all, I have to write a key. I’m going to use a class for that:

1
2
3
4
5
6
7
8
9
10
class Key {
	std::array<std::array<unsigned int, 3>, 3> data;

public:
	Key() {
		for (auto& row : data) {
			row.fill(0);
		}
	}
};

This is the most simple implementation of the key class. Of course, I can’t do anything here yet, but it can store 3x3 data AND it has a constructor that fills every cell of the matrix with 0.

I’ll need to add two functions: get_value and set_value. The first function will take two parameters: row and column in order to find the exact place of the number that I’ll be looking for. On the other hand, the second function will take three parameters: row, column and value. With these parameters I can tell the key where I want to store a number and what number I want to store:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
class Key {
	std::array<std::array<unsigned int, 3>, 3> data;

public:
	Key() {
		for (auto& row : data) {
			row.fill(0);
		}
	}

	bool set_value(unsigned int column, unsigned int row, unsigned int value) {
		if (column >= 0 && column < 3 && row >= 0 && row < 3) {
			data[column][row] = value;

			return true;
		}

		return false;
	}

	unsigned int get_value(unsigned int column, unsigned int row) {
		if (column >= 0 && column < 3 && row >= 0 && row < 3) {
			return data[column][row];
		}

		return 0;
	}
};

Alright. This should work. Now it’s time to implement a vector.

Writing vector class

Alright. The principle is the same here, but a vector is going to be a 1x3 matrix, not 3x3.

1
2
3
4
5
6
7
8
9
10
class Vector {
	std::array<unsigned int, 3> data;

public:
	Vector() {
		for (auto& cell : data) {
			cell = 0x00;
		}
	}
};

Ok, time to write get_value and set_value functions. BTW, there is going to be a difference with parameters. That’s because there is only one dimension, so I’m just going to use column:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
class Vector {
	std::array<unsigned int, 3> data;

public:
	Vector() {
		for (auto& cell : data) {
			cell = 0x00;
		}
	}

	bool set_value(unsigned int column, unsigned int value) {
		if (column >= 0 && column < 3) {
			data[column] = value;

			return true;
		}

		return false;
	}

	unsigned int get_value(unsigned int column) {
		if (column >= 0 && column < 3) {
			return data[column];
		}

		return 0;
	}
};

Adding character encoding

In the Hill Cipher, characters are represented by numbers. A is 0 and Z is 25. Before I add an actual character encoding, I’ll need to make little changes in the Key and Vector classes. In set_value function to be exact. I’m going to check value and if it’s not going to be [0-25], then I’m not going to write anything:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
class Key {
	std::array<std::array<unsigned int, 3>, 3> data;

public:
	Key() {
		for (auto& row : data) {
			row.fill(0);
		}
	}

	bool set_value(unsigned int column, unsigned int row, unsigned int value) {
		if (column >= 0 && column < 3 && row >= 0 && row < 3 && value >= 0 && value < 26) {
			data[column][row] = value;

			return true;
		}

		return false;
	}

	unsigned int get_value(unsigned int column, unsigned int row) {
		if (column >= 0 && column < 3 && row >= 0 && row < 3) {
			return data[column][row];
		}

		return 0;
	}
};

class Vector {
	std::array<unsigned int, 3> data;

public:
	Vector() {
		for (auto& cell : data) {
			cell = 0x00;
		}
	}

	bool set_value(unsigned int column, unsigned int value) {
		if (column >= 0 && column < 3 && value >= 0 && value < 26) {
			data[column] = value;

			return true;
		}

		return false;
	}

	unsigned int get_value(unsigned int column) {
		if (column >= 0 && column < 3) {
			return data[column];
		}

		return 0;
	}
};

Ok, classes are fine. Time to add character encoding. First of all, I’m going to create a class which will hold enum and two functions: encode_character and decode_character. The first function will receive char value and convert it to [0-25] unsigned integer format. On the other hand, the second function will receive a number that is going to be [0-25] unsigned integer format and output a character:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
class CharacterEncoder {
	enum Characters : int {
		A = 0, B = 1, C = 2, D = 3, E = 4, F = 5,
		G = 6, H = 7, I = 8, J = 9, K = 10, L = 11,
		M = 12, N = 13, O = 14, P = 15, Q = 16, R = 17,
		S = 18, T = 19, U = 20, V = 21, W = 22, X = 23,
		Y = 24, Z = 25
	};

public:
	CharacterEncoder() {

	}

	unsigned int encode_character(char character) {
		int index = character - 'A';

		if (index < 0 || index > 25) {
			return 0;
		}

		Characters number = static_cast<Characters>(index);

		return number;
	}

	char decode_character(unsigned int number) {
		if (number < 0 || number > 25) {
			return '#';
		}

		char ch = static_cast<char>(number + 'A');

		return ch;
	}
};

BTW, I return # symbol in decode_character if an error happens because the function HAS to return char value, so # is going to be a symbol of error and TERROR!!!

Alright. In the same class, I’ll write a function that encrypts and decrypts data. Here is encrypt_vector vector function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
std::array<unsigned int, 3> encrypt_vector(std::array<unsigned int, 3> vector, std::array<std::array<unsigned int, 3>, 3> key) {
	std::array<unsigned int, 3> result_vector = { 0, 0, 0 };

	for (unsigned int i = 0; i < 3; i++) {
		unsigned int product = 0;

		for(unsigned int j = 0; j < 3; j++){
			product += key[i][j] * vector[j];
			result_vector[j] = product;
		}
	}

	return result_vector;
}

There is one problem. I need to implement the calculation of the inverse matrix of the key. First of all, I’ll have to check if the key matrix is invertible, and if it is, then I can calculate the inverse matrix. I’m going to add verify_key function into Key class, that’ll verify that the key is valid(Inversible matrix). It turns out I need several functions to make this work. First of all, I need mod26 function to calculate modulo. I also need functions for calculating the determinant and calculating if inversing is possible, and finally, I’ll need a function that will inverse the key matrix:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
class Key {
	std::array<std::array<unsigned int, 3>, 3> data;

	int mod26(int val) {
		val %= 26;

		if (val < 0) {
			val += 26;
		}

		return val;
	}

	int determinant(std::array<std::array<unsigned int, 3>, 3> matrix) {
		return mod26(matrix[0][0] * (matrix[1][1] * matrix[2][2] - matrix[1][2] * matrix[2][1])
			- matrix[0][1] * (matrix[1][0] * matrix[2][2] - matrix[1][2] * matrix[2][0])
			+ matrix[0][2] * (matrix[1][0] * matrix[2][1] - matrix[1][1] * matrix[2][0]));
	}

	int inverse(int val) {
		val = mod26(val);
		for (unsigned int i = 1; i < 26; i++) {
			if ((val * i) % 26 == 1) {
				return i;
			}
		}

		return -1;
	}

	std::array<std::array<unsigned int, 3>, 3> invert_matrix(std::array<std::array<unsigned int, 3>, 3> matrix) {
		int l_determinant = determinant(matrix);
		int l_determinant_inverse = inverse(l_determinant);
		if (l_determinant_inverse == -1) {
			return { { {0, 0, 0}, {0, 0, 0}, {0, 0, 0} } };
		}

		std::array<std::array<unsigned int, 3>, 3> inverted_matrix;

		inverted_matrix[0][0] = mod26((matrix[1][1] * matrix[2][2] - matrix[1][2] * matrix[2][1]) * l_determinant_inverse);
		inverted_matrix[0][1] = mod26(-(matrix[0][1] * matrix[2][2] - matrix[0][2] * matrix[2][1]) * l_determinant_inverse);
		inverted_matrix[0][2] = mod26((matrix[0][1] * matrix[1][2] - matrix[0][2] * matrix[1][1]) * l_determinant_inverse);

		inverted_matrix[1][0] = mod26(-(matrix[1][0] * matrix[2][2] - matrix[1][2] * matrix[2][0]) * l_determinant_inverse);
		inverted_matrix[1][1] = mod26((matrix[0][0] * matrix[2][2] - matrix[0][2] * matrix[2][0]) * l_determinant_inverse);
		inverted_matrix[1][2] = mod26(-(matrix[0][0] * matrix[1][2] - matrix[0][2] * matrix[1][0]) * l_determinant_inverse);

		inverted_matrix[2][0] = mod26((matrix[1][0] * matrix[2][1] - matrix[1][1] * matrix[2][0]) * l_determinant_inverse);
		inverted_matrix[2][1] = mod26(-(matrix[0][0] * matrix[2][1] - matrix[0][1] * matrix[2][0]) * l_determinant_inverse);
		inverted_matrix[2][2] = mod26((matrix[0][0] * matrix[1][1] - matrix[0][1] * matrix[1][0]) * l_determinant_inverse);

		return inverted_matrix;
	}

public:
	Key() {
		for (auto& row : data) {
			row.fill(0);
		}
	}

	bool set_value(unsigned int column, unsigned int row, unsigned int value) {
		if (column >= 0 && column < 3 && row >= 0 && row < 3 && value >= 0 && value < 26) {
			data[column][row] = value;

			return true;
		}

		return false;
	}

	unsigned int get_value(unsigned int column, unsigned int row) {
		if (column >= 0 && column < 3 && row >= 0 && row < 3) {
			return data[column][row];
		}

		return 0;
	}

	bool verify_key() {
		std::array<std::array<unsigned int, 3>, 3> check = invert_matrix(data);
		std::array<std::array<unsigned int, 3>, 3> zero = { {{0,0,0},{0,0,0},{0,0,0}} };

		if (check == zero) {
			return false;
		}

		return true;
	}
};

Ok. This is Key class now. My head almost exploded three times when I was writing those calculations (finally, my mistakes were fixed by ChatGPT). I’m not a math professor, ALRIGHT???

I think the code is done here. I also have encrypt_vector and decrypt_vector functions ready. All the malware has to do now is to iterate over data and load characters slowly into a vector to do encryption or decryption. The final code is 190 lines long:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
#pragma once

#include <array>

class CharacterEncoder {
	enum Characters : int {
		A = 0, B = 1, C = 2, D = 3, E = 4, F = 5,
		G = 6, H = 7, I = 8, J = 9, K = 10, L = 11,
		M = 12, N = 13, O = 14, P = 15, Q = 16, R = 17,
		S = 18, T = 19, U = 20, V = 21, W = 22, X = 23,
		Y = 24, Z = 25
	};

public:
	CharacterEncoder() {

	}

	unsigned int encode_character(char character) {
		int index = character - 'A';

		if (index < 0 || index > 25) {
			return 0;
		}

		Characters number = static_cast<Characters>(index);

		return number;
	}

	char decode_character(unsigned int number) {
		if (number < 0 || number > 25) {
			return '#';
		}

		char ch = static_cast<char>(number + 'A');

		return ch;
	}

	std::array<unsigned int, 3> encrypt_vector(std::array<unsigned int, 3> vector, std::array<std::array<unsigned int, 3>, 3> key) {
		std::array<unsigned int, 3> result_vector = { 0, 0, 0 };

		for (unsigned int i = 0; i < 3; i++) {
			unsigned int product = 0;

			for(unsigned int j = 0; j < 3; j++){
				product += key[i][j] * vector[j];
				result_vector[j] = product;
			}
		}

		return result_vector;
	}

	std::array<unsigned int, 3> decrypt_vector(std::array<unsigned int, 3> vector, std::array<std::array<unsigned int, 3>, 3> inversed_key) {
		std::array<unsigned int, 3> result_vector = { 0, 0, 0 };

		for (unsigned int i = 0; i < 3; i++) {
			unsigned int product = 0;

			for (unsigned int j = 0; j < 3; j++) {
				product += inversed_key[i][j] * vector[j];
				result_vector[j] = product;
			}
		}

		return result_vector;
	}
};

class Key {
	std::array<std::array<unsigned int, 3>, 3> data;

	int mod26(int val) {
		val %= 26;

		if (val < 0) {
			val += 26;
		}

		return val;
	}

	int determinant(std::array<std::array<unsigned int, 3>, 3> matrix) {
		return mod26(matrix[0][0] * (matrix[1][1] * matrix[2][2] - matrix[1][2] * matrix[2][1])
			- matrix[0][1] * (matrix[1][0] * matrix[2][2] - matrix[1][2] * matrix[2][0])
			+ matrix[0][2] * (matrix[1][0] * matrix[2][1] - matrix[1][1] * matrix[2][0]));
	}

	int inverse(int val) {
		val = mod26(val);
		for (unsigned int i = 1; i < 26; i++) {
			if ((val * i) % 26 == 1) {
				return i;
			}
		}

		return -1;
	}

	std::array<std::array<unsigned int, 3>, 3> invert_matrix(std::array<std::array<unsigned int, 3>, 3> matrix) {
		int l_determinant = determinant(matrix);
		int l_determinant_inverse = inverse(l_determinant);
		if (l_determinant_inverse == -1) {
			return { { {0, 0, 0}, {0, 0, 0}, {0, 0, 0} } };
		}

		std::array<std::array<unsigned int, 3>, 3> inverted_matrix;

		inverted_matrix[0][0] = mod26((matrix[1][1] * matrix[2][2] - matrix[1][2] * matrix[2][1]) * l_determinant_inverse);
		inverted_matrix[0][1] = mod26(-(matrix[0][1] * matrix[2][2] - matrix[0][2] * matrix[2][1]) * l_determinant_inverse);
		inverted_matrix[0][2] = mod26((matrix[0][1] * matrix[1][2] - matrix[0][2] * matrix[1][1]) * l_determinant_inverse);

		inverted_matrix[1][0] = mod26(-(matrix[1][0] * matrix[2][2] - matrix[1][2] * matrix[2][0]) * l_determinant_inverse);
		inverted_matrix[1][1] = mod26((matrix[0][0] * matrix[2][2] - matrix[0][2] * matrix[2][0]) * l_determinant_inverse);
		inverted_matrix[1][2] = mod26(-(matrix[0][0] * matrix[1][2] - matrix[0][2] * matrix[1][0]) * l_determinant_inverse);

		inverted_matrix[2][0] = mod26((matrix[1][0] * matrix[2][1] - matrix[1][1] * matrix[2][0]) * l_determinant_inverse);
		inverted_matrix[2][1] = mod26(-(matrix[0][0] * matrix[2][1] - matrix[0][1] * matrix[2][0]) * l_determinant_inverse);
		inverted_matrix[2][2] = mod26((matrix[0][0] * matrix[1][1] - matrix[0][1] * matrix[1][0]) * l_determinant_inverse);

		return inverted_matrix;
	}

public:
	Key() {
		for (auto& row : data) {
			row.fill(0);
		}
	}

	bool set_value(unsigned int column, unsigned int row, unsigned int value) {
		if (column >= 0 && column < 3 && row >= 0 && row < 3 && value >= 0 && value < 26) {
			data[column][row] = value;

			return true;
		}

		return false;
	}

	unsigned int get_value(unsigned int column, unsigned int row) {
		if (column >= 0 && column < 3 && row >= 0 && row < 3) {
			return data[column][row];
		}

		return 0;
	}

	bool verify_key() {
		std::array<std::array<unsigned int, 3>, 3> check = invert_matrix(data);
		std::array<std::array<unsigned int, 3>, 3> zero = { {{0,0,0},{0,0,0},{0,0,0}} };

		if (check == zero) {
			return false;
		}

		return true;
	}
};

class Vector {
	std::array<unsigned int, 3> data;

public:
	Vector() {
		for (auto& cell : data) {
			cell = 0x00;
		}
	}

	bool set_value(unsigned int column, unsigned int value) {
		if (column >= 0 && column < 3 && value >= 0 && value < 26) {
			data[column] = value;

			return true;
		}

		return false;
	}

	unsigned int get_value(unsigned int column) {
		if (column >= 0 && column < 3) {
			return data[column];
		}

		return 0;
	}
};

Conclusion

Well, cryptography is fun. You can make up some really weird shit to obfuscate data. You might think, “Toko, could we use more simple encryption algos to encrypt the data?” Well, yes, but actually no. Complex mechanisms like this actually help malware to avoid AV engines because complex tasks take time and INSTRUCTIONS!!! To be honest, this will make malware more undetectable for AV engines than the simple XOR function. That’s because there is more non-violent code written, and MORE code means MORE time doing non-violent things. AV engines love that…

RELATED POSTS
Attacking Russian scam website (Part I)
Attacking Russian scam website (Part I)I know a guy who lives in the U.S., and he told me that he received a suspicious ...
Attacking Nov 07 11 min
How I hacked Russian guy with ADB Shell
How I hacked Russian guy with ADB ShellEveryone knows how the internet works. If you have anything open, you’ll most lik...
Attacking Dec 01 17 min
How I exploited Intel's Active Management Technology in China
How I exploited Intel’s Active Management Technology in ChinaIn previous blog, I talked about how I managed to exploit t...
Attacking Dec 02 6 min
Forcing dead 4C+ to boot (Part I)
Forcing “dead” Radxa 4C+ to boot (Part I)Several days ago I got my hands on Radxa 4C+ which is an alternative for Raspbe...
Radxa Oct 26 6 min
OffSec Squid game
Squid loves to pass stuff around…No exploits?This CTF was quite interesting because there were no exploit PoCs initially...
CTF Oct 14 9 min
OffSec ClamAV challenge
Mailing a plastic explosiveIn this challenge I’m going to mail a C4 plastic explosive to the host. When this explodes, i...
CTF Sep 30 8 min
View More Posts