Unsupported Screen Size: The viewport size is too small for the theme to render properly.

Import External Data — Failing For Table With 24m Rows

forums forums SQLyog Sync tools, Migration, Scheduled Backup and Notifications Import External Data — Failing For Table With 24m Rows

Tagged: 

This topic contains 5 replies, has 0 voices, and was last updated by  sathish 4 years, 10 months ago.

  • Author
    Posts
  • #13167

    I am attempting to import external data from and Oracle database.

     

    I am able to successfully execute an import with smaller tables, but when I try to execute a table or subset of a table with ~20M rows it fails. A windows pop-up screen opened and indicates the project has failed. The screen asked me to close the program or look for solutions online and close the program.

     

    The job log isn’t giving me much information around the issue. Are there other logs I can look at or are there other suggestions on how to get this working?

  • #34813

    Here is the problem details reported by windows:

     

    Problem signature:

      Problem Event Name: APPCRASH

      Application Name: SJA.exe

      Application Version: 11.3.3.0

      Application Timestamp: 52f47755

      Fault Module Name: StackHash_8902

      Fault Module Version: 6.1.7601.18247

      Fault Module Timestamp: 521eaf24

      Exception Code: c0000374

      Exception Offset: 00000000000c4102

      OS Version: 6.1.7601.2.1.0.274.10

      Locale ID: 1033

      Additional Information 1: 8902

      Additional Information 2: 89026bc74985bfcf0cb15871f8fe9a88

      Additional Information 3: d5f7

      Additional Information 4: d5f7254acdaad60e1c0fcae91c777c94

     

    Read our privacy statement online:

     

    If the online privacy statement is not available, please read our privacy statement offline:

      C:Windowssystem32en-USerofflps.txt
  • #34814

    sathish
    Member

    Hello,

     

    1) What is the operation system?

    2) The SQLyog version(32 bit or 64 bit)?

    3) What is the type of conencion(Direct/HTTP/SSH)?

    4) Could you please tell us if this is consistently reproducible?

    5) It would be helpful if you could provide us the structure of the table where this issue is reproducible. 

     

    Regards,

    Sathish

  • #34815

    sathish
    Member

    Hello,

     

    Also there is a known issue with import external data where ODBC/ANSI SQL does not have a CHUNK control which could cause memory exhaustion in either server or client at some point. We already have this in our issue tracker here:  https://code.google.com/p/sqlyog/issues/detail?id=1709 

     

    What you could do here is to have more jobs using a WHERE clause splitting that huge import into multiple smaller import jobs. 

     

    Please refer: https://static.webyog.com/docs/SQLyog/Migration_Toolkit_-_From_table-SQLWhere_SQLyog_MySQL_GUI.htm

     

    Regards,

    Sathish

  • #34816

    chuckhorton
    Member

    Any update on this bug? I am having the same problem. 

     

    Sathish: your link  https://code.google.com/p/sqlyog/issues/detail?id=1709  does not work.

  • #34817

    chuckhorton
    Member

    More info on my problem:

     

    When I run an import job using sja.exe I get APPCRASH Error. The first time I ran this job it worked. When I tried it again (several times over the last few hours) I get the APPCRASH error. It seems to crash on Importing table schema: PR_EMPLOYEE_MASTER_1_MC. This job has been running for years on another computer with old version of Sqlyog. 

     

    Version of SQLyog Ultimate – MySQL GUI v11.33 (64 bit)

    On Windows Server 2008 R2 Standard

     

    See below for details.
     

    Job started at Thu Mar 13 16:13:22 2014

     

    DBMS Information: MICROSOFT SQL SERVER

     

    Importing table schema: EC_EQUIPMENT_MASTER_MC… Successful…

    Importing table schema: JC_JOB_MASTER_MC… Successful…

    Importing table schema: JC_PHASE_MASTER_MC… Successful…

    Importing table schema: PR_EMPLOYEE_MASTER_1_MC…

    U:>

     

    Problem signature:

      Problem Event Name: APPCRASH

      Application Name: SJA.exe

      Application Version: 11.3.3.0

      Application Timestamp: 52f47755

      Fault Module Name: StackHash_2272

      Fault Module Version: 6.1.7601.17725

      Fault Module Timestamp: 4ec4aa8e

      Exception Code: c0000374

      Exception Offset: 00000000000c40f2

      OS Version: 6.1.7601.2.1.0.272.7

      Locale ID: 1033

      Additional Information 1: 2272

      Additional Information 2: 2272626ea2e743c653c456b9f85ac430

      Additional Information 3: 5156

      Additional Information 4: 5156add447adfe2ee682079f0c3c6779

     

    PR_EMPLOYEE_MASTER_1_MC – table structure

     

    Field

    Type

    Allow Null

    Default Value

    Company_Code

    varchar(3)

    No

     

    Employee_Code

    varchar(11)

    No

     

    Employee_Name

    varchar(30)

    No

     

    Alpha_Sort

    varchar(8)

    No

     

    Street_Address

    varchar(25)

    No

     

    Address

    varchar(25)

    No

     

    State

    varchar(2)

    No

     

    Zip_Code

    varchar(10)

    No

     

    Phone

    varchar(14)

    No

     

    Social_Security

    varchar(9)

    No

     

    Employee_Date_List1

    timestamp

    Yes

     

    Employee_Date_List2

    timestamp

    Yes

     

    Employee_Date_List3

    timestamp

    Yes

     

    Employee_Date_List4

    timestamp

    Yes

     

    Employee_Date_List5

    timestamp

    Yes

     

    Employee_Date_List6

    timestamp

    Yes

     

    Employee_Date_List7

    timestamp

    Yes

     

    Sex

    varchar(4)

    No

     

    Race

    varchar(4)

    No

     

    Marital_Status

    varchar(4)

    No

     

    Casual_Flag

    varchar(4)

    No

     

    Pay_Frequency_Flag

    varchar(4)

    No

     

    Pay_Type

    varchar(4)

    No

     

    Last_Pay_Type

    varchar(4)

    No

     

    Spouse_Name

    varchar(15)

    No

     

    Title

    varchar(20)

    No

     

    Supervisor_Code

    varchar(4)

    No

     

    Tax_Control_Flag_List1

    varchar(4)

    No

     

    Tax_Control_Flag_List2

    varchar(4)

    No

     

    Tax_Control_Flag_List3

    varchar(4)

    No

     

    Tax_Control_Flag_List4

    varchar(4)

    No

     

    Tax_Control_Flag_List5

    varchar(4)

    No

     

    Tax_Control_Flag_List6

    varchar(4)

    No

     

    Tax_Control_Flag_List7

    varchar(4)

    No

     

    Tax_Control_Flag_List8

    varchar(4)

    No

     

    Tax_Control_Flag_List9

    varchar(4)

    No

     

    Tax_Control_Flag_List10

    varchar(4)

    No

     

    Tax_Control_Flag_List11

    varchar(4)

    No

     

    Filing_Status_List1

    varchar(4)

    No

     

    Filing_Status_List2

    varchar(4)

    No

     

    Filing_Status_List3

    varchar(4)

    No

     

    Filing_Status_List4

    varchar(4)

    No

     

    Filing_Status_List5

    varchar(4)

    No

     

    Tax_Table_Codes_List1

    varchar(4)

    No

     

    Tax_Table_Codes_List2

    varchar(4)

    No

     

    Tax_Table_Codes_List3

    varchar(4)

    No

     

    Tax_Table_Codes_List4

    varchar(4)

    No

     

    Tax_Table_Codes_List5

    varchar(4)

    No

     

    Number_Of_Exemptions_List1

    decimal(12,2)

    No

    0.00

    Number_Of_Exemptions_List2

    decimal(12,2)

    No

    0.00

    Number_Of_Exemptions_List3

    decimal(12,2)

    No

    0.00

    Number_Of_Exemptions_List4

    decimal(12,2)

    No

    0.00

    Number_Of_Exemptions_List5

    decimal(12,2)

    No

    0.00

    Tax_Add_List1

    decimal(12,2)

    No

    0.00

    Tax_Add_List2

    decimal(12,2)

    No

    0.00

    Tax_Add_List3

    decimal(12,2)

    No

    0.00

    Tax_Add_List4

    decimal(12,2)

    No

    0.00

    Tax_Add_List5

    decimal(12,2)

    No

    0.00

    Tax_Amount_Control_List1

    varchar(3)

    No

     

    Tax_Amount_Control_List2

    varchar(3)

    No

     

    Tax_Amount_Control_List3

    varchar(3)

    No

     

    Tax_Amount_Control_List4

    varchar(3)

    No

     

    Tax_Amount_Control_List5

    varchar(3)

    No

     

    Delete_Flag

    varchar(3)

    No

     

    Employment_Status

    varchar(3)

    No

     

    Minimum_Earnings

    decimal(12,2)

    No

    0.00

    Number_Of_Dependents

    decimal(12,2)

    No

    0.00

    Worker_Comp_Code

    varchar(6)

    No

     

    Bank_Account_Number

    varchar(17)

    No

     

    Driver_License

    varchar(25)

    No

     

    Last_Pay_Rate

    decimal(12,2)

    No

    0.00

    Hourly_Rate

    decimal(12,4)

    No

    0.0000

    Salary_Amount

    decimal(12,2)

    No

    0.00

    Annual_Pay_Rate

    decimal(12,2)

    No

    0.00

    Special_Pay_Rate

    decimal(12,2)

    No

    0.00

    Special_Pay_Amount

    decimal(12,2)

    No

    0.00

    Meal_Allowance_Rate

    decimal(12,2)

    No

    0.00

    Department_Code

    varchar(6)

    No

     

    Selected_Flag

    varchar(3)

    No

     

    Months_To_Next_Review

    decimal(12,2)

    No

    0.00

    Emergency_Contact

    varchar(25)

    No

     

    Emergency_Phone

    varchar(14)

    No

     

    Emergency_Relationship

    varchar(15)

    No

     

    Not_Used1

    decimal(12,2)

    No

    0.00

    Not_Used2

    decimal(12,2)

    No

    0.00

    Not_Used3

    decimal(12,2)

    No

    0.00

    Hours_List1

    decimal(12,2)

    No

    0.00

    Hours_List2

    decimal(12,2)

    No

    0.00

    Hours_List3

    decimal(12,2)

    No

    0.00

    Standard_Work_Hours

    decimal(12,2)

    No

    0.00

    Update_Flag

    varchar(3)

    No

     

    Union_Code

    varchar(6)

    No

     

    Total_Hours

    decimal(12,2)

    No

    0.00

    Check_Type

    varchar(3)

    No

     

    Manual_Check_Number

    decimal(12,2)

    No

    0.00

    OT_Hours

    decimal(12,2)

    No

    0.00

    Domicile_Address

    varchar(25)

    No

     

    Domicile_City

    varchar(25)

    No

     

    Domicile_State

    varchar(3)

    No

     

    Domicile_Zip_Code

    varchar(10)

    No

     

    Pay_Rate_Code

    varchar(3)

    No

     

    Employee_Work_Class

    varchar(3)

    No

     

    ABA_Number

    varchar(9)

    No

     

    Bank_Account_Type

    varchar(3)

    No

     

    Last_Job_Number

    varchar(10)

    No

     

    Last_Equipment_Code

    varchar(10)

    No

     

    Wage_Class

    varchar(3)

    No

     

    Certified_Flag

    varchar(3)

    No

     

    Billing_Code

    varchar(3)

    No

     

    I9_Flag

    varchar(3)

    No

     

    I9_Expire_Date

    timestamp

    Yes

     

    EEO_Class_Code

    varchar(4)

    No

     

    Stop_Tax_Flag_List1

    varchar(4)

    No

     

    Stop_Tax_Flag_List2

    varchar(4)

    No

     

    Stop_Tax_Flag_List3

    varchar(4)

    No

     

    Stop_Tax_Flag_List4

    varchar(4)

    No

     

    Calc_Flag

    varchar(4)

    No

     

    Auto_Deposit_Flag

    varchar(4)

    No

     

    Burden_Percent

    decimal(12,2)

    No

    0.00

    Suppress_Last_Job_Flag

    varchar(3)

    No

     

    Prevail_Wage_Fringe_Rate

    decimal(12,2)

    No

    0.00

    Pay_Rate_Percent

    decimal(12,2)

    No

    0.00

    ABA_Number_2

    varchar(9)

    No

     

    ABA_Number_3

    varchar(9)

    No

     

    ABA_Number_4

    varchar(9)

    No

     

    ABA_Number_5

    varchar(9)

    No

     

    Bank_Account_Type_2

    varchar(3)

    No

     

    Bank_Account_Type_3

    varchar(3)

    No

     

    Bank_Account_Type_4

    varchar(3)

    No

     

    Bank_Account_Type_5

    varchar(3)

    No

     

    Bank_Account_Number_2

    varchar(17)

    No

     

    Bank_Account_Number_3

    varchar(17)

    No

     

    Bank_Account_Number_4

    varchar(17)

    No

     

    Bank_Account_Number_5

    varchar(17)

    No

     

    Auto_Deposit_Flag_2

    varchar(3)

    No

     

    Auto_Deposit_Flag_3

    varchar(3)

    No

     

    Auto_Deposit_Flag_4

    varchar(3)

    No

     

    Auto_Deposit_Flag_5

    varchar(3)

    No

     

    Auto_Deposit_Allocation

    varchar(3)

    No

     

    Auto_Deposit_Rate1

    decimal(12,2)

    No

    0.00

    Auto_Deposit_Rate2

    decimal(12,2)

    No

    0.00

    Auto_Deposit_Rate3

    decimal(12,2)

    No

    0.00

    Auto_Deposit_Rate4

    decimal(12,2)

    No

    0.00

    Auto_Deposit_Rate5

    decimal(12,2)

    No

    0.00

    First_Name

    varchar(20)

    No

     

    Middle_Name

    varchar(20)

    No

     

    Last_Name

    varchar(30)

    No

     

    Occupation

    varchar(20)

    No

     

    Trade

    varchar(20)

    No

     

    Street_Address2

    varchar(25)

    No

     

    Domicile_Address2

    varchar(25)

    No

     

    Effective_Date

    timestamp

    Yes

     

    Comments

    varchar(40)

    No

     

    Last_Crew

    varchar(10)

    No

     

    Cost_Center

    varchar(10)

    No

  • #34818

    sathish
    Member

    The link had two extra spaces at the end. Here is the correct one: https://code.google.com/p/sqlyog/issues/detail?id=1709

You must be logged in to reply to this topic.