Creating large files very slow

Message Topic Search Topic Options Post Reply Create New Topic Printable Version Translate Topic

   We use QuickPDF to generate laboratory reports.  On average, these run about 500 pages, but recently we've had a few large projects that resulted in 3000+ page reports.  Probably 2000 of these pages consist of embedded graphics.

The report generation seems to go by fairly quickly, considering the amount of data that needs to be pulled together and streamed over the network.  However, when we issue the QP.SaveToFile(FFileName) it takes another 30-40 minutes to actually create the file.  This is on a 4.2GHz Athlon 64 X2.  I don't have access to the sources, so I don't know what happens in the SaveToFile method, but I thought that the PDF was assembled in memory and that the SaveToFile routine just copied that data out to disk.  If this were the case, it should only take a couple of seconds at most (64MB file size).  Can somebody point me to what may be happening here, and is there anything I can do to speed this up?

Thanks

Author	Message Topic Search Topic Options Post Reply Create New Topic Printable Version Translate Topic
mrgcms Members Profile Find Members Posts Beginner Joined: 27 Sep 06 Location: United States Status: Offline Points: 9	Post Options Post Reply Quote mrgcms Report Post Thanks(0) Quote Reply Topic: Creating large files very slow Posted: 20 Nov 07 at 9:57PM
	We use QuickPDF to generate laboratory reports. On average, these run about 500 pages, but recently we've had a few large projects that resulted in 3000+ page reports. Probably 2000 of these pages consist of embedded graphics. The report generation seems to go by fairly quickly, considering the amount of data that needs to be pulled together and streamed over the network. However, when we issue the QP.SaveToFile(FFileName) it takes another 30-40 minutes to actually create the file. This is on a 4.2GHz Athlon 64 X2. I don't have access to the sources, so I don't know what happens in the SaveToFile method, but I thought that the PDF was assembled in memory and that the SaveToFile routine just copied that data out to disk. If this were the case, it should only take a couple of seconds at most (64MB file size). Can somebody point me to what may be happening here, and is there anything I can do to speed this up? Thanks

Ingo Members Profile Find Members Posts Moderator Group Joined: 29 Oct 05 Status: Offline Points: 3524	Post Options Post Reply Quote Ingo Report Post Thanks(0) Quote Reply Posted: 21 Nov 07 at 4:34PM
	Hi! Memory and time is often a problem working on big pdf files with quickpdf. It seems to me that in many code parts memory won't resetted ... I'm using something like this: QP := TiSEDQuickPDF.Create; try QP.UnlockKey('mycode'); dafh := QP.DAOpenFile(edit,''); x := QP.DAGetPageCount(dafh); STR := ''; verztxt := edit + '.txt'; AssignFile(cf,verztxt); Rewrite(cf); pc := 0; for i := 1 to x Do begin dapr := QP.DAFindPage(dafh,i); QP.CombineLayers; STR := QP.DAExtractPageText(dafh,dapr,3); WriteLn(cf,Trim(STR)); pc := pc + 1; if ( pc = 10 ) Then begin pc := 0; QP.DACloseFile(dafh); QP.Free; QP := TiSEDQuickPDF.Create; QP.UnlockKey('mycode'); dafh := QP.DAOpenFile(edit,''); end; // . . . This works fine on big files for textextraction - no problems anymore for me. After few pages i'm using "free" and then a new "creation" ... This needs only less time and avoid memory-exceptions for me. Best regards, Ingo

marian_pascalau Members Profile Find Members Posts Debenu Quick PDF Library Expert Joined: 28 Mar 06 Location: Germany Status: Offline Points: 278	Post Options Post Reply Quote marian_pascalau Report Post Thanks(0) Quote Reply Posted: 22 Nov 07 at 4:57AM
	Hi Dave, you should send a test sample to support(at)quickpdf.org.

mrgcms Members Profile Find Members Posts Beginner Joined: 27 Sep 06 Location: United States Status: Offline Points: 9	Post Options Post Reply Quote mrgcms Report Post Thanks(0) Quote Reply Posted: 22 Nov 07 at 1:53PM
	Hi Marian, Unfortunately, it's in the report generation that this occurs, after I've assembled the 3,000 pages in memory. I'll put together a test app that uses a single graphics page that's repeated (I don't think QuickPDF is 'intelligent' enough to share these) and I'll see if it's still slow. If so, I'll upload a test project. It's Thanksgiving over her now, so it may be Monday before I go back into the office and put this together. Thanks

Michel_K17 Members Profile Find Members Posts Newbie www.exp-systems.com Joined: 25 Jan 03 Status: Offline Points: 297	Post Options Post Reply Quote Michel_K17 Report Post Thanks(0) Quote Reply Posted: 23 Nov 07 at 9:27AM
	Things I would check: RAM: How large if the final report (in MB) and how much memory is in your PC?. Might it be large enough that Windows has run out of room in it's RAM and having to rely on the PageFile as extra memory? NETWORK: Is it significantly faster if you save on the local HD rather than over the network? Happy Thanksgiving. Drove 12 hours on Wed so that I could do the family (in-laws) get-together last night... Ate too much. :) Cheers!
	Michel

mrgcms Members Profile Find Members Posts Beginner Joined: 27 Sep 06 Location: United States Status: Offline Points: 9	Post Options Post Reply Quote mrgcms Report Post Thanks(0) Quote Reply Posted: 04 Dec 07 at 1:33PM
	Final report is only about 16MB System has 1GB RAM and an Athlon 64 X2 4.2GHz system I've just repeated with the file being saved to the local drive without any improvement in speed. We're at end of year crunch right now, but I'm trying to find time to put together a test program that'll demonstrate this.