Debenu Quick PDF Library - PDF SDK Community Forum : Textextraction: Determine the used codepage

Textextraction: Determine the used codepage : Hi,here a short adwise how make...

Thu, 29 Dec 2011 19:53:09 +0000

Author: edvoigt
Subject: 2078
Posted: 29 Dec 11 at 7:53PM

Hi,

here a short adwise how make it not so dirty and more sure.

The solution above determes if a font is using afii-codes. Consequently you have to figure out, in which part of the PDF-Text is which font used. The test only if a such font exists is rather unexact. It may happen, there is a mixture of languages (and codepages) on one page. This is rather sure if there are parts from different PDFs combined to a new one. Therefore there is a journey first through the fontdefinitions and than through the content needed.

Werner

Textextraction: Determine the used codepage : Hi,it seems to be a good source...

Fri, 23 Dec 2011 13:41:50 +0000

Author: edvoigt
Subject: 2078
Posted: 23 Dec 11 at 1:41PM

Hi,

it seems to be a good source here:

http://www.science.co.il/language/locale-codes.asp?s=codepage

There are some other forwarding links.

And you need the relation between afii-codes and copepage.

May it helps.

Werner

Textextraction: Determine the used codepage : HiWerner!Thanks a lot for this!Now...

Fri, 23 Dec 2011 12:53:28 +0000

Author: Ingo
Subject: 2078
Posted: 23 Dec 11 at 12:53PM

Hi Werner!

Thanks a lot for this!
Now i can go further ...
So there's the question which are the most used worldwide encode-formats ;-)

To you and all the other ones here

a Merry Christmas and a Happy New Year,

Ingo

Textextraction: Determine the used codepage : Hi Ingo, Hi Andrew,the goal...

Fri, 23 Dec 2011 11:22:44 +0000

Author: edvoigt
Subject: 2078
Posted: 23 Dec 11 at 11:22AM

Hi Ingo, Hi Andrew,

the goal is not to paint a glyph correctly, therefore it is enough to figure out the use of the codepage 1251 with its charset.

Here a quick&dirty solution without to much work:

function IsQP1251(fn: string): boolean;
var
QP: TQuickPDF;
obj: string;
i, n, glyphno, error, p: integer;
begin
Result := false;
QP := TQuickPDF.Create;
if QP.UnlockKey({$I PDFkey.inc}) = 1 // 8.xx
then begin
    QP.LoadFromFile(fn, '');
    n := QP.GetObjectCount;
    i := 1;
    repeat    // search for an PDF-object /Encoding with afii-codes for cyrillic inside
      obj := QP.GetObjectToString(i); // afii10017-afii10846
      if (Pos('/Encoding', obj)>0) and (Pos('/Differences', obj)>0)
      then begin
        p := Pos('afii10', obj);
      if (p>0)
        then begin       // 17..846
          Val(copy(obj, p+6, 3), glyphno, error);
        Result := (error=0) and (glyphno>=17) and (glyphno<=84 6);
        end;
      end;
      inc(i);
    until (i>n) or Result;
end;
end;

This works under Delphi7 with QuickPDF 8.13

The basic idea is:

inside the PDF is an encoding-object, which may start so:
/Type /Encoding /BaseEncoding /WinAnsiEncoding /Differences
So I go over all objects and look for /Encoding and /Differences.

The /Differences-array is the key. There are strings starting with "afii" followed by a number. They denote a special interpretation of the value of a byte.

It is dirty, because I am not sure about is there ever a such /Encoding-entry in the PDF.

It is dirty, because I go the short way directly to an object. Better to go along through the tree (/Procset => /Font => /Encoding), but the afii-codes are rather unique in conjunction with /Differences.

I dont know, may this Encoding-object be compressed too? In this case it would be more work, another reason for my wish (case 9605) for a GetInflatedObjectTo... as great brother of GetObjectTo... used in the solution above.

An overview you may get here
http://partners.adobe.com/public/developer/en/opentype/aglfn13.txt

A good source is
http://www.adobe.com/content/dam/Adobe/en/devnet/font/pdfs/5013.Cyrillic_Font_Spec.pdf
too.

The solution works with the two examples I got from Ingo. Maybe there is more to do. But it is a first entry.

Cheers and merry christmas

Werner

Edited by edvoigt - 23 Dec 11 at 12:42PM

Textextraction: Determine the used codepage : Hi Werner, Hi Andrew! I begin...

Fri, 23 Dec 2011 08:13:04 +0000

Author: Ingo
Subject: 2078
Posted: 23 Dec 11 at 8:13AM

Hi Werner, Hi Andrew!

I begin to realize ... really not an easy job ;-)

I have to read ... Thank you both for the links.

Cheers, Ingo

Textextraction: Determine the used codepage : Extracting text is not an easy...

Fri, 23 Dec 2011 07:59:07 +0000

Author: AndrewC
Subject: 2078
Posted: 23 Dec 11 at 7:59AM

Extracting text is not an easy process. Each font can either be defined as single byte or multi byte font and they also contain an Encoding entry in the CMap which is similar to a code page. To make things more complex a font can also contain a ToUnicode mapping array to help with text extraction routines. On top of that a font can contain a Differences array which can remap any character code to a new code.

Also some documents contains subsetted fonts which allow you remap any character code to any other code and this can make text extraction impossible as you can tell the font to draw a 'A' but the font actually draws as a 'B'.

QPL hides this functionality to make text extraction easy to use. GetPageText options 3 - 8 are more advanced than option 0 and 1.

The only way to understand what is going on is to look at the CMaps contained in each font.

http://www.adobe.com/content/dam/Adobe/en/devnet/font/pdfs/5014.CIDFont_Spec.pdf

Andrew

Textextraction: Determine the used codepage : Hi Werner!You've got an...

Thu, 22 Dec 2011 22:31:52 +0000

Author: Ingo
Subject: 2078
Posted: 22 Dec 11 at 10:31PM

Hi Werner!

You've got an email ... right now ;-)
Thanks in advance.

Cheers, Ingo

Edited by Ingo - 22 Dec 11 at 10:32PM

Textextraction: Determine the used codepage : Hi Ingo,the PDF-Spec gives no...

Thu, 22 Dec 2011 22:19:53 +0000

Author: edvoigt
Subject: 2078
Posted: 22 Dec 11 at 10:19PM

Hi Ingo,

the PDF-Spec gives no Results if I search for "codepage". Therefore it is not so simple. But there is an explanation in 5.5.5 Character Encoding. I hope this is the key.

To verify this, it is necessary to have a longer look in an inflated encoding dictionary and an entry in it. For a such test a PDF with codepage 1251 is needed.

This means a longer way thru some objects. Can you mail me a such PDF?

Erläuterungen natürlich besser für mich auf deutsch.

Cheers,

Werner

Textextraction: Determine the used codepage : Hi! Now myself ... ;-) I'm...

Thu, 22 Dec 2011 15:36:37 +0000

Author: Ingo
Subject: 2078
Posted: 22 Dec 11 at 3:36PM

Hi!

Now myself ... ;-)

I'm an extensive user of the extract functionality still i'm mostly working with version 7.26 together with Delphi 2007 and Delphi 5.

Actually i'm trying to get the textcontents as unicode.

I can extract the text made in arabic, russian and many other languages as long as they were created with utf8 and after extraction i'll get it all with ...

ansistring := utf8decode(widestring)

As samples i've many foreign pdf-documents. There are few russian documents, too.

Few i can extract 'cause they were made with utf8 - other russian documents failed 'cause they were made with codepage 1251.

My questions now:

Is there a functionality available to detect the used codepage in a textcontent in an automated way?

How to decode codepage 1251 - something similar to utf8decode?

Hope someone can help me out.

Thanks a lot in advance.

Cheers, Merry Christmas and a Happy New Year to all of you

Ingo

Edited by Ingo - 22 Dec 11 at 3:39PM