[Cuis-dev] Non-ASCII characters in source code files Was: TrueType font import problems

Thu Sep 14 06:16:16 PDT 2023

Hi Juan,

Thanks for the explanation. Did I understand correctly that old ISO-8859-15-encoded source files should be converted to UTF-8 before they are loaded in Unicode-enabled Cuis? Or does Cuis somehow do that automatically if possible?

IIUC the files on https://github.com/Cuis-Smalltalk/StyledTextEditor should have been converted to UTF-8 already. If yes, I still don't understand why the string literal in RTFExporting.pck.st was changed by resaving the package file? (I did not use my repo for the test.)

How did you do the conversion? Did you use some external tool? (I could not find any code for this in Cuis except from UnicodeString>>#fromBytesStream: but there are no senders.)

Regarding the method iso8859s15ToRTFEncoding, I am pretty sure this is the correct string from the comment (Test for Cent and Euro characters):
self assert: 'A¢€' iso8859s15ToRTFEncoding = 'A\u162?\u8364?'

Instead of #iso8859s15ToRTFEncoding a new method #toRTFEncoding or #asRTF polymorphic to String and UnicodeString is probably needed, right?

Cheers,
Bernhard

------- Original Message -------
Juan Vuletich via Cuis-dev <cuis-dev at lists.cuis.st> schrieb am Donnerstag, 14. September 2023 um 13:33:

> 
> 
> Hi Bernhard,
> 
> We have two unrelated issues.
> 
> 1)- Yes, I removed TextFontFamilyAndSize. When I fixed loading of
> Optima.tcc I reviewed font handling in general, and I thought it would
> be more comfortable to users to control Font ans Size as two separate
> attributes. So now we have TextFontFamily and TextFontSize. I fixed the
> editors and any tests for the base image. I forgot about
> StyledTextEditor. Apologies. Any code that was aware of
> TextFontFamilyAndSize now needs to be aware of TextFontFamily and
> TextFontSize. Changes are relatively straightforward.
> 
> 2)- Non-ASCII characters in source code files. Until about one year ago,
> Cuis didn't care much about Unicode. Files were saved in ISO-8859-15
> encoding. This means ASCII [0..127] and a slight modification of Latin-1
> [160..255], and some hand picked Math symbols in [128..159]. Around one
> year ago I added full support for Unicode in Cuis. This also meant
> converting all files to UTF-8 encoding, where non-ASCII characters use
> more than one byte. There are very few places where this makes a
> difference, and you found a few in those tests. I'm pretty sure I was
> careful enough to do those conversions correctly, including
> StyledTextEditor. So, I think that whatever is at
> https://github.com/Cuis-Smalltalk/StyledTextEditor is correct. Maybe
> you'd need to update your fork. In any case, a review for correctness is
> more than welcome.
> 
> Hope this helps.
> 
> Thanks,
> 
> 
> On 9/14/2023 6:38 AM, Bernhard Pieber via Cuis-dev wrote:
> 
> > Hi Juan,
> > 
> > Thanks for the fix. It works as expected.
> > 
> > I found the fonts improvement broke loading StyledTextEditor, though, because it still references TextFontFamilyAndSize which was removed. I guess this might also affect other packages?
> > 
> > I prepared a branch which fixes it without breaking additional tests:
> > https://github.com/bpieber/Cuis-Smalltalk-StyledTextEditor/tree/fix-STE
> > 
> > I wanted to create a PR but then saw that there are some changes I did not do on purpose. You can see an example here in line 10:
> > https://github.com/bpieber/Cuis-Smalltalk-StyledTextEditor/commit/0b58898b75167454db54298db0803cd2c75c4156
> > 
> > This is how it is shown in SourceTree:
> > !String methodsFor: '*rtfExporting' stamp: 'jmv 9/5/2016 20:27:00'!
> > iso8859s15ToRTFEncoding
> > "Convert the given string to RTF escaped Unicode from the internal encoding: ISO Latin 9 (ISO 8859-15)"
> > "
> > - self assert: ('A¢§' iso8859s15ToRTFEncoding) hex = ' 'A\u162?\u8364?''
> > + self assert: ('A¬¢‚Ç¨' iso8859s15ToRTFEncoding) hex = ' 'A\u162?\u8364?''
> > "
> > 
> > In the terminal it looks still a bit different:
> > git show 0b58898 -- RTFExporting.pck.st
> > 
> > !String methodsFor: '*rtfExporting' stamp: 'jmv 9/5/2016 20:27:00'!
> > iso8859s15ToRTFEncoding
> > "Convert the given string to RTF escaped Unicode from the internal encoding: ISO Latin 9 (ISO 8859-15)"
> > "
> > - self assert: ('A<A2><A4>' iso8859s15ToRTFEncoding) hex = ' 'A\u162?\u8364?''
> > + self assert: ('A¢€' iso8859s15ToRTFEncoding) hex = ' 'A\u162?\u8364?''
> > 
> > I could reproduce the behaviour like this:
> > 1. Start with the latest image and install updates.
> > 2. Checkout the master branch of StyledTextEditor.
> > 3. Temporaryly recreate class TextFontFamilyAndSize.
> > 4. Feature require: 'StyledText'.
> > 5. Open Installed Packages.
> > 6. Click on RTFExporting.
> > 7. Click save.
> > 
> > I think the problem already happened in step 4 when Cuis was reading the package file. Saving in step 7 makes it apparent.
> > 
> > This type of behaviour happened in other places as well. See line 710 here:
> > https://github.com/bpieber/Cuis-Smalltalk-StyledTextEditor/commit/c1d67b56ffb707e189d144b696c31ee8f3d2bf21#diff-fc4d98b72cc240e23f1fbf7486239a158bcb860548588a80825fe6abf3f0b6a0
> > 
> > And I am afraid it also happened in past changes which are already in master. See e.g. line 853 here:
> > https://github.com/bpieber/Cuis-Smalltalk-StyledTextEditor/commit/57ce26012da883def313cbf005a6ac60081545e8#diff-ef2065cf0d5ae47c54dfc101485dcdd7a4f02c4a91eb164f6cfa4e1cdc2e442c
> > 
> > Why does this happen? Are the changes correct or erroneous?
> > 
> > Cheers,
> > Bernhard
> > 
> > ------- Original Message -------
> > Juan Vuletich via Cuis-devcuis-dev at lists.cuis.st schrieb am Dienstag, 12. September 2023 um 16:20:
> > 
> > > Fix now at GitHub.
> > > 
> > > Thanks!
> > > 
> > > On 9/11/2023 5:07 PM, Bernhard Pieber via Cuis-dev wrote:
> > > 
> > > > Hi Juan,
> > > > 
> > > > Thanks for improving this. I had problems getting it to work, though.
> > > > 
> > > > At first I tried this:
> > > > TrueTypeFontFamily readTrueTypeFontEntry: '/System/Library/Fonts' asDirectoryEntry // 'Optima.ttc'.
> > > > 
> > > > However, this method still has the old behavior: Optima ExtraBlack replaces Optima Regular as the baseTTFontDescription.
> > > > 
> > > > After some digging I found I needed to copy Optima.ttc to its own directory and use this method instead:
> > > > TrueTypeFontFamily readAllTrueTypeFontsIn: DirectoryEntry smalltalkImageDirectory / 'TrueTypeFonts' / 'Optima'.
> > > > 
> > > > This works as you described. Now I two families Optima and Optima-ExtraBlack.
> > > > 
> > > > The method TrueTypeFontFamily<<#readTrueTypeFontEntry: is used when installing a font file from a file list, so it should probably be fixed as well.
> > > > 
> > > > Cheers,
> > > > Bernhard
> > > > 
> > > > ------- Original Message -------
> > > > Juan Vuletichjuan at cuis.st schrieb am Mittwoch, 30. August 2023 um 01:00:
> > > > 
> > > > > Hi Bernhard,
> > > > > 
> > > > > I pushed your fix to support V2 headers.
> > > > > 
> > > > > But I also did a few more things. When importing fonts like Optima,
> > > > > ExtraBlack and other styles beyond regular / italic / bold / bold italic
> > > > > will be turned into different FontFamilies.
> > > > > 
> > > > > I also changed the handling of fonts and sizes in TextEditors quite a
> > > > > bit. I think it is nicer now.
> > > > > 
> > > > > Please pull, update, and test.
> > > > > 
> > > > > Thanks,
> > > > > 
> > > > > On 8/27/2023 5:23 AM, Bernhard Pieber via Cuis-dev wrote:
> > > > > 
> > > > > > Here is a slightly improved version.
> > > > > > 
> > > > > > > Am 26.08.2023 um 20:59 schrieb Bernhard Pieber via Cuis-devcuis-dev at lists.cuis.st:
> > > > > > > 
> > > > > > > Hi everyone,
> > > > > > > 
> > > > > > > I tried importing a TrueType font I had on my system (Optima.ttc). I copied the file to a new folder named Fonts and used the following:
> > > > > > > TrueTypeFontFamily readAllTrueTypeFontsIn: DirectoryEntry currentDirectory / 'Fonts'
> > > > > > > 
> > > > > > > This resulted in the error "This TTF Collection has an unsupported header version“ in TTFontReader>>#readTTCFontFrom:.
> > > > > > > 
> > > > > > > The reason is that the font uses a TTC Header Version 2.0, see:
> > > > > > > https://docs.fileformat.com/font/ttc/
> > > > > > > 
> > > > > > > I changed the code to allow both header versions. If I understand the difference between version 1 and 2 of the header formats correctly, this should be a safe change. Installing the font worked with this change.
> > > > > > > 
> > > > > > > However, instead of Optima Regular it used Optima ExtraBlack when I selected Optima as the font. I found no way to select Optimal Regular as the font style.
> > > > > > > 
> > > > > > > Optima.ttc contains five styles:
> > > > > > > Optima Regular
> > > > > > > Optima Italic
> > > > > > > Optima Bold
> > > > > > > Optima Bold Italic
> > > > > > > Optima ExtraBlack
> > > > > > > 
> > > > > > > The code for importing the TTC file imports all 5 styles:
> > > > > > > TTFontReader readTTCFrom: (DirectoryEntry currentDirectory / 'Fonts' // 'Optima.ttc') binaryContents.
> > > > > > > 
> > > > > > > However, only 4 of those 5 are saved in the font family:
> > > > > > > TrueTypeFontFamily availableFamilies at: 'Optima'
> > > > > > > 
> > > > > > > There seems to be a restriction TrueTypeFontFamily that there can only be one font variant per TextEmphasis combination.
> > > > > > > 
> > > > > > > As a workaround I changed TTFontDescription>>#emphasis to identify Black as Bold. Now Optima Regular is used as default. However, I cannot use Optima Bold now.
> > > > > > > 
> > > > > > > I attach the ChangeSet with the fix and the workaround.
> > > > > > > 
> > > > > > > Given that we use only TrueType fonts now, I think the best solution would be to remove bold an italic from TextEmphasis and use the all variants of the TrueType font families instead.
> > > > > > > 
> > > > > > > By the way, is there a package with a nice font viewer?
> > > > > > > 
> > > > > > > Cheers,
> > > > > > > Bernhard
> > > > > > > --
> > > > > > > Cuis-dev mailing list
> > > > > > > Cuis-dev at lists.cuis.st
> > > > > > > https://lists.cuis.st/mailman/listinfo/cuis-dev
> > > > > > > <5984-CuisCore-BernhardPieber-2023Aug26-15h22m-bp.001.cs.st>
> > > > > > > --
> > > > > > > Juan Vuletich
> > > > > > > cuis.st
> > > > > > > github.com/jvuletich
> > > > > > > researchgate.net/profile/Juan-Vuletich
> > > > > > > independent.academia.edu/JuanVuletich
> > > > > > > patents.justia.com/inventor/juan-manuel-vuletich
> > > > > > > linkedin.com/in/juan-vuletich-75611b3
> > > > > > > twitter.com/JuanVuletich
> > > 
> > > --
> > > Juan Vuletich
> > > cuis.st
> > > github.com/jvuletich
> > > researchgate.net/profile/Juan-Vuletich
> > > independent.academia.edu/JuanVuletich
> > > patents.justia.com/inventor/juan-manuel-vuletich
> > > linkedin.com/in/juan-vuletich-75611b3
> > > twitter.com/JuanVuletich
> > > 
> > > --
> > > Cuis-dev mailing list
> > > Cuis-dev at lists.cuis.st
> > > https://lists.cuis.st/mailman/listinfo/cuis-dev
> 
> 
> 
> --
> Juan Vuletich
> cuis.st
> github.com/jvuletich
> researchgate.net/profile/Juan-Vuletich
> independent.academia.edu/JuanVuletich
> patents.justia.com/inventor/juan-manuel-vuletich
> linkedin.com/in/juan-vuletich-75611b3
> twitter.com/JuanVuletich
> 
> --
> Cuis-dev mailing list
> Cuis-dev at lists.cuis.st
> https://lists.cuis.st/mailman/listinfo/cuis-dev