Memory Issues in Talend while working with large files

Question

I have a job which uses a 3GB input file using Talend Open Studio for Big Data. Below is the screenshot of my job.

But on the first execution of this job, it gave me an error related to heap memory issue.

On the second run, it gave me an error related to garbage collector error.

And now, it's not showing error, but simply executing and doing nothing and finally stopping.

Can someone explain why I am getting this kind of error?

Answer 1

I think the problem lies in the huge amount of data that is being returned by the tSampleRow component. Try using a where clause in your tMysqlInput component in order to filter and fetch less number of rows. Also, in the advanced settings select the “stream” option in order to stream the rows.

Hope this was what you were looking for. 

Get your Talend certification today to become certified professional.

Thanks.