Migration Toolkit Halt

[Newbie23]

I am using SQLyog Enterprise (i'm a new user), and I am having difficulty with uisng the migration toolkit.

I am trying to import about 20 tables, some with 1 to 8 million records. The data are coming from CSV files.

Individual tables and data seem to come in just fine, but when I try to have several of the tables imported in one session, SQLyog just stops processing records without completing the importation. This usually occurs after at least a few hundred thousand records, but not clearly at the same location. The tables are therefore not complete, and the remainder of the tables to be imported are not processed at all. The 'NEXT' button in the wizard is enabled, and if pressed, I get a message that the importation occurred sucessfully, though it is apparent if did not.

There are no error messages, and nothing indicating a problem in the sja log file. The same problem occurs if 'Abort on Error' is checked or not. CPU activity stops.

I suppose this could be a MySQL timeout issue, but I get no indication of this from MySql Administrator.

Sorry for the long post, and my apologies if I placed this in the wrong section.

Thank you.

[peterlaursen]

Yes it could very much look like a time-out thing. Or at least some configuration issue.

Just one question: why do you use CSV-files with the Migration Tool. Can't you conncet to the ODBC-database directly? You use the TEXT-ODBC-driver then?

This

Quote:

Individual tables and data seem to come in just fine, but when I try to have several of the tables imported in one session, SQLyog just stops processing records without completing the importation

would not occur then, since then one table is finished before the next is started.

If you need to use CSV-files you can import from CSV without using the Migration Toolkit. Tried that?

What is the original database for the data ?

And the MySQL version ?

[peterlaursen]

@Newbie23

It also would be nice to know whether the MySQL serveris a local server or a server at an ISP – and if the latter is the case – whether tunnelling is used. If for instance you use HTTP-tunnelling, it could be a php-issue.

Actually all that we ask for here http://www.webyog.com/faq/6_26_en.htmlwould be helpful if we shall be able to reproduce the problem.

[Newbie23]

Sorry for the delay in the response and thank for you for your assistance. My answers are listed below.

* Program (SQLyog) version (version 5.0)

* MySQL server version(s) (version 5.0)

* Connection type (direct, HTTP or SSH tunnelling) (direct)

* Storage engine(s) – especially when not MyISAM (had similar problem with both MyIsam and InnoDb)

* The platform on which the server runs (WinXP pro)

* Any unusual server configuration (be that a special character set, ANSI-mode or whatever). ( Nothing that I can identify)

I am not sure of the original database manager for the data … I 'suspect' it is Oracle.

On further testing, I can get a reproducible error (similar halt in execution) in one of the tables after about 6 million records imported (out of a little over 9 milliion). When I import directly from the data files (ie within SQLyog), the entire table imports without difficulty.

I would prefer to use the migration tool kit because:

1. I have about 20 CSV tables to import at a time. (and done monthly)

2. On occasion, the data structure may change and I do not necessarily get advance notice. I like how SQLyog migration toolkit maps data and allows modification.

I could send a screen dump, but it really does not list anything except the listing of records imported stopping at about 6 million, and the next button being enabled.

Thanks again for the assistance!!!

[Ritesh]

Could you please send us the CSV dump and the associated table structure? Please send it to [email protected].

We will destroy the data once we are done with the test.

[Ritesh]

I think we have found the issue. Actually, there is no way in ODBC to get chunks of data as we can do in MySQL using LIMIT syntax.

So we actually do a complete SELECT * FROM TABLE, to get all the data and load them in the memory. It's not a memory leakage issue in SQLyog for sure because even Access takes up that much memory to import that same data.

The strange thing is that it works with 4.1 and not with 4.2 BETA onwards when there is no code change in the Migration Toolkit module except for support for table spaces in Oracle and Sybase.

I am looking into it and I believe we can get a solution within the next 24 hours.

[Author]

Posts