get the sourcecode [of UTF-8]

[A bughunter]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

My reply to Otto is interspersed.
Originating Question

Where to get the sourcecode of relevent (version) UTF-8?: in order to checksum text against the specific encoding map (codepage).

from A_bughunter at proton.me

Sent with Proton Mail secure email.

On Wednesday, November 6th, 2024 at 17:19, Otto Stolz <otto.stolz at uni-konstanz.de> wrote:

> Hello bughunter,

I did look over the FAQ and did not find an answer to this question. Definitions and terms are not a problem as I layed them out in my second post you would see in the mailing list archive here https://corp.unicode.org/pipermail/unicode/2024-November/011102.html
> before wording a question to any discussion group, it is recommended
> to read (and understand) the pertinent FAQ list; otherwise the ensuing
> discussion will focus on definitions and terms rather than the problem
> at hand. You may start reading at https://www.unicode.org/faq/.
It would seem the only problem we have with Definitions & Terms is others' inability to read and speak English. I furthered that Definitions & Terms are not a problem in my reply to Jim here https://corp.unicode.org/pipermail/unicode/2024-November/011117.html

The question of mine "Where to get the sourcecode" is  absolutely clear because if I install a Debian FOSS only system producing UTF-8 texts then there must be sourcode. The problem (of checksums)  which may not be clear to 'You' is only mentioned as a use-case and is not fully onpoint of this mailing list. Focus on this question.
> That said, I’ll try to answer your question. As your problem is not
> quiet clear, you’ll get basically three answers, and a technical hint
> pertaining to two of them.
> 
> You have asked::
> 
> > Where to get the sourcecode of relevent (version) UTF-8?:
> > in order to checksum text against the specific encoding map (codepage).
> 
More about my use-case you will find my statement " In order to fully authenticate: the codepage of the character to glyph map must be known." in my second post here https://corp.unicode.org/pipermail/unicode/2024-November/011102.html the use-case is certainly worth mentioning because "in order to"- the source must be known. I restated for Slawomir to focus on My maxim in my reply to Slawomir here https://corp.unicode.org/pipermail/unicode/2024-November/011111.html saying " the only part you need to focus on to answer the originating question is: "the character to glyph map must be known."" because my query to the mailing list a prerequisit for my use-case. I showed the use-case is intended only to accentuate the need to answer the question. 


What I do with the sourcecode or how I will proceed to checksum text has no bearing on answering: "Where to get the sourcecode".
> My answer depends on the purpose of the checksum.
> 
> UTF-8 is one method (of a handfull of standardized methods) to
> represent Unicode text at the bit level in order to conveniently
> transfer, or store, it. If the intend of your checksum is merely
> to protect against transmission error, or tempering, then you
> would simply checksum this bit-level representation of the text –
> no knowledge of Unicode, or UTFs, is required to achieve this goal.
> 
Protect against corruption and tampering are two fine points for the use of checksums. However for you to say  "If my intent is against tampering no knowledge of UTF is required" is not quite true. And I can carry on this discussion with you on my GitHub page here https://github.com/freedom-foundation

> A Unicode code point is a number in the range from 0 to 1 114 111;
> a Unicode text is a sequence of Unicode code points.
> On the bit level, you can represent that sequence in various ways,
> cf. https://www.unicode.org/faq/utf_bom.html. Hence, if you
> 
> want to compare two Unicode texts that are represented in arbitrary
> bit-level representations (UTFs), then you would convert those
> to the same UTF (preferably UTF-32) and checksum those. (UTF-32
> stores the 21 bits needed to represent a Unicode code point in
> one 32 bit wide storage location, leaving 11 bits unused.)
 
Yet another cause of why one need's the sourcecode of the system encoding the UTF-8: "In Unicode, some characters may be represented in various ways" and this is why I did mention the use-case of checksum text'
> In Unicode, some characters may be represented in various ways;
> e. g. an “é” can be coded as one single Unicode code point, viz.
> U+00E9, LATIN SMALL LETTER E WITH ACUTE, or, alternatively, as
> a pair of Unicode code points, viz. U+0065 U+0301 LATIN SMALL
> LETTER E + COMBINING ACUTE ACCENT. To cope with ambiguities of
> this kind, Unicode defines those two representations as
> “canonically equivalent”, i. e., they are to be treated in every
> respect as equivalent and interchangeable, for details,
> cf. https://www.unicode.org/faq/normalization.html. Hence,
> 

> if you want to check that two Unicode texts are canonically
> equivalent, you would first convert them to UTF-32, then
> ‘normalize’ them (i. e. choose consistently the same representation
> for all instances of canonically equivalent encodings), then
> checksum the normalized representations.

Yes I had noticed ICU however I need to look at whatever is actually being used. However ICU is sidelined and does not apply to what the users are actually using.
> You were asking for source code, but the better way to do
> conversion and normalizations is by using an established and
> well-tested program library, such as ICU,
> cf. https://icu.unicode.org/#h.i33fakvpjb7o.
 
You seem so eager to tackle the extra problem. Keep in mind the checksum problem is "extra unicode" meaning it is outside of the scope of the Unicode consortium mailing list. My query here is a sideline to my GitHub repo Unicode_map you may see here https://github.com/freedom-foundation/unicode_map and you may see my Proof of Concept https://github.com/freedom-foundation/unicode_map?tab=readme-ov-file#proof-of-concept if you would like to discuss the fine points of checksumming 
> Good luck with your project,
> Otto
-----BEGIN PGP SIGNATURE-----
Version: ProtonMail

wnUEARYKACcFgmcsKd4JkKkWZTlQrvKZFiEEZlQIBcAycZ2lO9z2qRZlOVCu
8pkAAOnHAQDWuVPE6hwEEMJrQrxAhUMsG//MOlpeCLAh7f2WfJ/WWwD+KaIt
zv26ZfCHgIIAkYfGJ1INIn4bq2wVxAruJnvW7Qo=
=qXm7
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: publickey - A_bughunter at proton.me - 0x66540805.asc
Type: application/pgp-keys
Size: 653 bytes
Desc: not available
URL: <https://corp.unicode.org/pipermail/unicode/attachments/20241107/ffd64e11/attachment.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: publickey - A_bughunter at proton.me - 0x66540805.asc.sig
Type: application/pgp-signature
Size: 119 bytes
Desc: not available
URL: <https://corp.unicode.org/pipermail/unicode/attachments/20241107/ffd64e11/attachment.sig>