NYCPHP Meetup

NYPHP.org

[nycphp-talk] OT: Standard Data Import Tool?

Matt Juszczak matt at atopia.net
Tue Dec 29 12:28:27 EST 2009


Hi all,

I need to write a tool that somehow takes multiple data feeds (some xls 
files, some csv files, etc.), defines each format, 
verifies/validates/sanitizes the data, and imports it into standardized 
MySQL tables.

As an example, one feed might look like this:

First Name, Last Name, DOB

And another like this:

Name, DOB

And another:

DOB, Last Name, First Name, Middle Initial

I need to somehow be able to say "For this source, this column needs to be 
split into two, and this other column is the date of birth column".  Once 
the data is parsed, it would need to be imported into standard MySQL 
tables:

first_name, last_name, birth_date, etc.

This is a simple example - the files and tables actually have 20-30 
columns each, so using mysqlimport or a simple shell script probably isn't 
the best option.

Are there open source tools out there that do this already?  If not, 
should I just build something?

Thanks,

-Matt



More information about the talk mailing list