gh-148252: Avoid overflow in _remote_debugging binary reader bounds checks

[maurycy]

I don't think a test makes sense; it'd fail only on 32-bit machines.

More details:

• _remote_debugging: prevent 32-bit integer overflow when decoding stack data from .pyb files #148252 (comment)

• Issue: _remote_debugging: prevent 32-bit integer overflow when decoding stack data from .pyb files #148252

[maurycy]

There's already SAMPLE_HEADER_FIXED_SIZE:

• https://github.com/python/cpython/blob/665b7dfcfa2/Modules/_remote_debugging/binary_io_writer.c#L26

but it's hard-coded 13. Maybe it should be changed to this addition?

grep for 13 yields more results:

• https://github.com/python/cpython/blob/665b7dfcfa2/Modules/_remote_debugging/binary_io_writer.c#L656
• https://github.com/python/cpython/blob/665b7dfcfa2/Modules/_remote_debugging/binary_io_writer.c#L657
• https://github.com/python/cpython/blob/665b7dfcfa2/Modules/_remote_debugging/binary_io_reader.c#L979

It looks that there's opportunity to move it to binary_io.h. Makes sense, @pablogsal ?

[pablogsal]

I think it makes sense indeed 👍 Can you do the change in this PR?

[maurycy]

@pablogsal

Voilà: f14741f

Truth to be told, I think there's a massive room for improvement with consts here:

https://github.com/maurycy/cpython/blob/remote-debugging-int-bounds/Modules/_remote_debugging/binary_io_reader.c#L26
https://github.com/maurycy/cpython/blob/remote-debugging-int-bounds/Modules/_remote_debugging/binary_io_reader.c#L26

cpython/Modules/_remote_debugging/binary_io_reader.c

Line 33 in b2f126c

#define MAX_DECOMPRESS_SIZE (1ULL << 30)

cpython/Modules/_remote_debugging/binary_io_reader.c

Line 133 in b2f126c

#define MAX_DECOMPRESS_SIZE (1ULL << 30)

or magic offsets:

cpython/Modules/_remote_debugging/binary_io_writer.c

Lines 1146 to 1154 in b2f126c

	uint64_t file_size = (uint64_t)footer_offset + 32;
	uint8_t footer[32] = {0};
	/* Cast size_t to uint32_t before memcpy to ensure correct bytes are copied
	* on both little-endian and big-endian systems (size_t is 8 bytes on 64-bit) */
	uint32_t string_count_u32 = (uint32_t)writer->string_count;
	uint32_t frame_count_u32 = (uint32_t)writer->frame_count;
	memcpy(footer + 0, &string_count_u32, 4);
	memcpy(footer + 4, &frame_count_u32, 4);
	memcpy(footer + 8, &file_size, 8);

(perhaps just FILE_FOOTER_SIZE here etc.)

https://github.com/maurycy/cpython/blob/f14741f422d0983f9a3c6a7bdd9c47b931e3fb1e/Modules/_remote_debugging/binary_io_reader.c#L985-L1003
https://github.com/maurycy/cpython/blob/f14741f422d0983f9a3c6a7bdd9c47b931e3fb1e/Modules/_remote_debugging/binary_io_writer.c#L651-L662
https://github.com/maurycy/cpython/blob/f14741f422d0983f9a3c6a7bdd9c47b931e3fb1e/Modules/_remote_debugging/binary_io_writer.c#L590-L595

(perhaps just SAMPLE_OFF warranted here as well)

etc., etc.

If interest in cleaning up these – I can create a PR.