NYCPHP Meetup

[Anthony Wlodarski]

I would have to agree.  The disk I/O is going to be large for a such a data set.  I think in this instance it is better to ask a few hundred bigger questions of the database then a few million smaller questions.

-Anthony


--
Anthony Wlodarski
www.thrillist.com <http://www.thrillist.com>
Web Applications Developer
568 Broadway Ste. 605
New York, NY, 10012
(o) 646.786.1944


________________________________
From: Jesse Callaway <bonsaime at gmail.com>
Reply-To: NYPHP Talk <talk at lists.nyphp.org>
Date: Fri, 13 Feb 2009 12:37:06 -0800
To: NYPHP Talk <talk at lists.nyphp.org>
Subject: Re: [nycphp-talk] slow php

On Fri, Feb 13, 2009 at 3:31 PM, Nicholas Hart <nhart at partsauthority.com> wrote:
> I have a php 'cron job' script which is inserting some records.  It uses
> arrays to store 12 field records along with several nested loops in which it
> tests various currency values from a separate 4 field table and sorts them
> to get the lowest one for insert into a new 4 field table record.  It has
> about 640,000 Product records per 4 Customers or apporx. 2.4 million records
> to insert.  It takes several hours+ to complete and I am hoping there is a
> way to speed it up.
>
> So far, I've tried updating instead of inserting records and changing a few
> functions but nothing seems to make much difference.  Am I being
> unreasonable for the size of the files here?  I'm considering using a
> Derived Table in mySQL and will also consider another language such as
> python.  Please let me know if you have any suggestions.  Thanks.
>
> Nicholas Hart
>
>
>
>
>
> _______________________________________________
> New York PHP User Group Community Talk Mailing List
> http://lists.nyphp.org/mailman/listinfo/talk
>
> http://www.nyphp.org/show_participation.php
>

Do your inserts in chunks. If you have the RAM, do one big insert.
Otherwise you have to hit the disk for each record which can be slow
especially if you have table locks going on elsewhere.

-jesse
_______________________________________________
New York PHP User Group Community Talk Mailing List
http://lists.nyphp.org/mailman/listinfo/talk

http://www.nyphp.org/show_participation.php

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nyphp.org/pipermail/talk/attachments/20090213/6810151d/attachment.html>