Hi George. Yeap, row by row is the way we have to...

2010-08-04T09:21:18.013+01:00

Hi George. Yeap, row by row is the way we have to do things. We deal with data from many heterogeneous systems that don't export data in the exact same ways. This data requires sanitizing and lookups into internal hash tables for their respective keys. Also some rows require actions to be performed i.e. new student or staff records may require new user accounts creating. We also need to derive certain data due to limitations of the external systems.

Using the MySQL "LOAD DATA INFILE" functionality was one of my first attempts(many years ago), which doesn't require any temp tables. We grew out of this method very quickly. If the data you take in is known to be clean and matches with your own systems schema then by all means use something like this. We don't have that luxury :)

I know this is an old post, but I thought it was w...

2010-08-03T22:48:53.625+01:00

I know this is an old post, but I thought it was worth a quick comment anyhow :)

Why are you performing individual inserts and updates? Would it not be more efficient to bulk load the CSV(s) in to a temporary table and then use a SET based method instead of an RBAR one?

Comments on Andrew Rose: MySQL: Unlimited updates in one query - ish.

Hi George. Yeap, row by row is the way we have to...

I know this is an old post, but I thought it was w...