text extraction

Message Topic Search Topic Options Post Reply Create New Topic Printable Version Translate Topic

   Hi,
at first, hello and greetings from Germany.
We have been using the Quick PDF library for many years and have solved all tasks well. But now I have a problem for which I cannot find a solution.

The problem is text extraction. 
With TTF fonts I get with the function 

GetTextBlockText(_TBID, _TBCounter)) 

I always get hieroglyphics like

T㠀䂎 쁰䀀䂏㴷砄␋Ȩ䁓T쳤視﫸〶�⻈ă徆㙺ăă抙㙺ăe㠀䂎 쁰䀀䂏㴷砄␋Ȩ䁓 䂀 䂀eࣘ〨튴밁젼視﫸〶�⻈ă徆㙺ăă抙㙺ăă抪㙺ăă搮㙺ăꚐ㯐Ꙡ㯐x㠀䂎 쁰䀀䂏㿰㴷砄␋Ȩ䁓 䂎 䁼 䂎x㯘쳤밁﫸〶⻍ă徆㙺ăă抙㙺ăă抪㙺ăă搮㙺ăꚐ㯐Ꙡ㯐�⻈⻍t㠀䂎 쁰䀀䂏㿰㴷砄␋Ȩ䁓 䂎 䁼␀䂖t㯘튴밁꼤륫﫸〶⻍ă徆㙺ăă抙

If I get the whole text of the page with 

GetPageText(0)

I get the correct text

"Text"

Can anyone help me?

regards

Peter

Author	Message Topic Search Topic Options Post Reply Create New Topic Printable Version Translate Topic
BAULOG Members Profile Find Members Posts Beginner Joined: 22 Feb 22 Location: Münster Status: Offline Points: 4	Post Options Post Reply Quote BAULOG Report Post Thanks(0) Quote Reply Topic: text extraction Posted: 22 Feb 22 at 1:33PM
	Hi, at first, hello and greetings from Germany. We have been using the Quick PDF library for many years and have solved all tasks well. But now I have a problem for which I cannot find a solution. The problem is text extraction. With TTF fonts I get with the function GetTextBlockText(_TBID, _TBCounter)) I always get hieroglyphics like T㠀䂎 쁰䀀䂏㴷砄␋Ȩ䁓T쳤視﫸〶�⻈ă徆㙺ăă抙㙺ăe㠀䂎 쁰䀀䂏㴷砄␋Ȩ䁓䂀䂀eࣘ〨튴밁젼視﫸〶�⻈ă徆㙺ăă抙㙺ăă抪㙺ăă搮㙺ăꚐ㯐Ꙡ㯐x㠀䂎 쁰䀀䂏㿰㴷砄␋Ȩ䁓䂎䁼䂎x㯘쳤밁﫸〶⻍ă徆㙺ăă抙㙺ăă抪㙺ăă搮㙺ăꚐ㯐Ꙡ㯐�⻈⻍t㠀䂎 쁰䀀䂏㿰㴷砄␋Ȩ䁓䂎䁼␀䂖t㯘튴밁꼤륫﫸〶⻍ă徆㙺ăă抙 If I get the whole text of the page with GetPageText(0) I get the correct text "Text" Can anyone help me? regards Peter

Ingo Members Profile Find Members Posts Moderator Group Joined: 29 Oct 05 Status: Offline Points: 3524	Post Options Post Reply Quote Ingo Report Post Thanks(0) Quote Reply Posted: 22 Feb 22 at 8:48PM
	Hi Peter :) It looks as if a Decrypt is missing after load and before extraction? BTW: GetTextBlockText needs an ExtractPageTextBlocks first - what's the result of this function? Perhaps a longer, relevant code snippet makes it easier to say more... Cheers and welcome here, Ingo (living near Bremen ;-)
	Cheers, Ingo

Ingo Members Profile Find Members Posts Moderator Group Joined: 29 Oct 05 Status: Offline Points: 3524	Post Options Post Reply Quote Ingo Report Post Thanks(0) Quote Reply Posted: 22 Feb 22 at 8:49PM
	aditional hint: A CombineContentStream and NormalizePage can help as well...
	Cheers, Ingo

BAULOG Members Profile Find Members Posts Beginner Joined: 22 Feb 22 Location: Münster Status: Offline Points: 4	Post Options Post Reply Quote BAULOG Report Post Thanks(0) Quote Reply Posted: 23 Feb 22 at 8:02AM
	Hi Ingo, thank you very much for the quick answer. Should we stick with English, as perhaps other users should also benefit from it? Here the code snippet. I have omitted the variable declarations (all started with "_"). PDFLib is the reference to the Debenu dll. If FileOpen(_File,_ID) _PageCount = PDFLib.PageCount() For _PageCounter = 1 To _ToPage _PCounter += 1 _TextCounter = 0 PDFLib.SelectPage(_PageCounter) PDFLib.NormalizePage(0) _TBID = PDFLib.ExtractPageTextBlocks(3) _TBCount = PDFLib.GetTextBlockCount(_TBID) For _TBCounter = 1 To _TBCount _TextList.add(PDFLib.GetTextBlockText(_TBID, _TBCounter)) Next Next FileClose(_ID) End If The PDF file was generated from a CAD application. All TTF texts (Arial etc.) cannot be read correctly. The SHX fonts (CAD specific) are read correctly. I would like to attach the pdf file. But I can't find a way to do it. many Thanks Peter Edited by BAULOG - 23 Feb 22 at 8:05AM

Ingo Members Profile Find Members Posts Moderator Group Joined: 29 Oct 05 Status: Offline Points: 3524	Post Options Post Reply Quote Ingo Report Post Thanks(0) Quote Reply Posted: 27 Feb 22 at 8:20PM
	Hi Peter, perhaps the pdf is encrypted? I don't see a decryption in your code... There are free online hoster offering a bit space combined with advertisement. You can use it with your sample-pdfs and post the link here.
	Cheers, Ingo

tfrost Members Profile Find Members Posts Senior Member Joined: 06 Sep 10 Location: UK Status: Offline Points: 437	Post Options Post Reply Quote tfrost Report Post Thanks(0) Quote Reply Posted: 28 Feb 22 at 2:04PM
	Your "hieroglyphics" look to me like 'mojibake', which is the name for Japanese (or Chinese) characters appearing when you mix up ASCII and DBCS. Have you checked in a debugger that you are not accidentally assigning the wrong character size somewhere?

BAULOG Members Profile Find Members Posts Beginner Joined: 22 Feb 22 Location: Münster Status: Offline Points: 4	Post Options Post Reply Quote BAULOG Report Post Thanks(0) Quote Reply Posted: 08 Mar 22 at 1:04PM
	Hi, at the following link you find a sample file with 4 lines text. Each text has a different font. I only get the first line as an readable text. https://cloud.baulog.de/index.php/s/4cSjGXdLXfcZqeJ Maybe someone else has an idea. Thanks for your help. Peter

tfrost Members Profile Find Members Posts Senior Member Joined: 06 Sep 10 Location: UK Status: Offline Points: 437	Post Options Post Reply Quote tfrost Report Post Thanks(0) Quote Reply Posted: 08 Mar 22 at 3:35PM
	I think it's a bug (which I can reproduce) in GetTextBlockText. After each valid character in the 130 characters returned for each (5-character) line of your text, there is a block of garbage 8-bit characters which form the mojibake string. This happens in every text line except your first one. Dropping all characters not in 0..9,a..z,A..Z reveals the correct results. It looks as if GetPageText(0) sanitizes the output somehow, to get rid of all the garbage. But it is hard to tell from the source where it does this, because the primary filter in this function is the area covered, and the filtering on content must then happen below this. My (ancient) copy of Debenu PDF Tools Pro must use this function, because it also displays all the text correctly when extracting. Interestingly, I opened your PDF in Affinity Publisher (which can edit PDFs), and found that Text1 completely disappears, whereas the other three are clear and editable, in substituted fonts Consolas, Tahoma and Arial respectively.

Ingo Members Profile Find Members Posts Moderator Group Joined: 29 Oct 05 Status: Offline Points: 3524	Post Options Post Reply Quote Ingo Report Post Thanks(0) Quote Reply Posted: 08 Mar 22 at 10:14PM
	I've got the same described issues with normal GetPageText with option 7 and option 3! My sample looks like this: --- page 1 from 1 --- "Arial";000000;0;234/118;"Text1" "AAAAAA+ISOCPEUR";000000;105;234/241;"T㠀䂎 쁰䀀䂏㎀ӷ툠彯切䁚T�ĕὈˑ糌˔ꯣ}꼹}e㠀䂎 쁰䀀䂏㎀ӷ툠彯切䁚䂀䂀eꄸӲ�ĕ艔ĕὈˑ糌˔ꯣ}꼹}꽊}냎}궠Ӯ군Ӯx㠀䂎 쁰䀀䂏㎀ӷ툠彯切䁚　䁼㠀䂎x鏨Ӻ萴ĉ분đὈˑ糌˔ꯣ}꼹}꽊}냎}궠Ӯ군Ӯ῜ˑt㠀䂎 쁰䀀䂏㎀ӷ툠彯切䁚　䁼⠀䂖t׭脴Č戌ĎὈˑ糌˔ꯣ}꼹}꽊}냎}궠Ӯ군Ӯ῜ˑ2㠀䂎 쁰䀀䂏㎀ӷ툠彯切䁚　䁸" "AAAAAA+Tahoma";000000;93;234/369;"T⠀䂐　쁺䀀䂏㎀ӷ氠穸咥䁗T脴Č׭耼˔ꯣ}꼹}e⠀䂐　쁺䀀䂏㎀ӷ氠穸咥䁗䀀䂂䀀䂂eꄸӲ脴Č矌Č׭耼˔ꯣ}꼹}꽊}냎}궠Ӯ군Ӯx⠀䂐　쁺䀀䂏㎀ӷ氠穸咥䁗瀀䂀堀䂑x鏨Ӻ脴Č鈌Ď׭耼˔ꯣ}꼹}꽊}냎}궠Ӯ군ӮὌˑt⠀䂐　쁺䀀䂏㎀ӷ氠穸咥䁗䁾᐀䂙t׭脴Č戌Ď׭耼˔ꯣ}꼹}꽊}냎}궠Ӯ군ӮὌˑ3⠀䂐　쁺䀀䂏㎀ӷ氠穸咥䁗䁴" "AAAAAA+ArialMT";000000;95;234/497;"T䀀䂐倀쁴䀀䂏㎀ӷ鄀ﭾ뀺䁗T矌ĕ䋸ˑ׭갦}꼹}e䀀䂐倀쁴䀀䂏㿰㎀ӷ鄀ﭾ뀺䁗က䂃᠀䂃က䂃eꄸӲ职ĕ䋸ˑ׭갦}꼹}꽊}냎}궠Ӯ군Ӯx䀀䂐倀쁴䀀䂏㿰㎀ӷ鄀ﭾ뀺䁗က䂃怀䂁㠀䂒x鏨Ӻ脴Č职ĕ䋸ˑ׭갦}꼹}꽊}냎}궠Ӯ군Ӯ׭׭t䀀䂐倀쁴䀀䂏뿰㎀ӷ鄀ﭾ뀺䁗ఀ䂚䀀䁿ఀ䂚tৈ׭脴Č䋸ˑ׭갦}꼹}꽊}냎}궠Ӯ군Ӯ׭׭4䀀䂐倀쁴䀀䂏㿰㎀ӷ鄀ﭾ뀺䁗怀䂞怀䁱 "
	Cheers, Ingo

tfrost Members Profile Find Members Posts Senior Member Joined: 06 Sep 10 Location: UK Status: Offline Points: 437	Post Options Post Reply Quote tfrost Report Post Thanks(0) Quote Reply Posted: 09 Mar 22 at 12:10PM
	Just to save others the trouble, I found that the DA versions of these functions produce the same garbage output, although the source code for the top-level functions is completely different. The winsoft.sk PDFIUM wrapper has an extract text sample which extracts all four of your examples correctly. It is available for Delphi or .Net but I have only tested the former. I feel able to mention it here because of the QPDF EOL announcement and the slim chance of this being improved in QPDF.