Jump to content


Photo

Data Sync For Huge Table


  • Please log in to reply
2 replies to this topic

#1 Andy Wong

Andy Wong

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 20 October 2011 - 07:47 AM

We're evaluation SQLyog as our data sync tool for a BI project. But when trying to sync a huge table (>10M rows) for the first time, it generates an error 2008 (MySQL client ran out of memory). I searched this forum and found a similar problem with a previous version but we're using the latest 9.30 which is supposed to be fixed.

Here's the log for your reference:

Sync started at Thu Oct 20 14:52:17 2011

`gc_txlog` 11099577 0 Error No. 2008
MySQL client ran out of memory


Our environment:
Source Server: Windows 2003 server 32-bit
Source Database: MySQL 5.0.26-community-nt
Target Server: Linux
Target Database: MySQL 5.0.77

Any idea? Thanks!

#2 ashwin

ashwin

    Advanced Member

  • Members
  • PipPipPip
  • 275 posts
  • Gender:Male
  • Location:Bangalore, India

Posted 20 October 2011 - 08:28 AM

Hi Andy Wong,

I searched this forum and found a similar problem with a previous version but we're using the latest 9.30 which is supposed to be fixed.


Please read 8.62 release notes:
http://www.webyog.co...-8-62-released/

"* SJA now supports an additional -r parameter that tells how big CHUNKS should be when copying to an empty table. The -r parameter has only effect with Data Sync jobs and is ignored with other types of jobs. This is a command line option only that is not supported from the GUI Wizard."

Also read FAQ: http://webyog.com/fa...agent_-sja.html

Quote from the FAQ: "Data sync jobs additionally supports a '-r' parameter (it is ignored with other types of jobs). It has only an effect when a non-empty source-table is synced to an empty target-table and defines how big CHUNKS should be fetched from source server (for a (source) HTTP connection it is 1000 rows by default if -r option is not specified explicitly). -r"2000" will copy CHUNKS of 2000 rows from source. If no -r parameter is specified SQLyog will fetch "all rows" from source server in one 'SELECT' query operation what may cause memory exhaustion on the client machine."

Please tell if you understand how to use this information or if you need more help.

Regards,
Ashwin A

#3 peterlaursen

peterlaursen

    Advanced Member

  • Admin
  • PipPipPip
  • 7,869 posts
  • Gender:Male
  • Location:Skagen, Denmark
  • Interests:well ... jazz/folk music, photography, chess, nature, ecology, history, bicycling, Highland Malts ... well, Lowland Malts and Cognac too actually :-) just wonder how I get the time to touch a computer! SQLyog and MONyog? no that's not interest, that's BASIC NEEDS simply!

Posted 20 October 2011 - 10:39 AM

Additional remark: this will happen only when you are syncing to an empty target. You may use the 'copy database to other host'/'copy table to other database' tool to populate for the first time.

DATA SYNC is not really intended to be a *copy tool*. It is a *sync tool* and the scenario with an empty target is not the main scope. But we did add the -R parameter to handle such situations even with large tables.
Computers make your grey hair come off ....

Peter Laursen
Webyog




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users