[nycphp-talk] slow php (slow database)
Paul A Houle
paul at devonianfarm.com
Fri Feb 13 16:43:44 EST 2009
The fastest way to get data into mysql is like this:
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
I needed to load 2.5M RDF triples into mysql to look up some
identifiers. My RDF store balked at the turtle file, so I hacked away
the header and treated it like a space-separated file. LOAD DATA INFILE
was scary fast... It loaded the data in 3 seconds!
Once you've loaded raw data into mysql you can do all kinds of tricks like
INSERT INTO ... SELECT
to process the information into the desired form. If you can do this
processing w/ clever mysql coding, you can probably do it in 1/10 the
LOC and more than 10 times the speed that it would take you in a
procedural language. The main thing is that you'll need to be
comfortable with joins, subselects and writing extreme queries.
More information about the talk
mailing list