"What is the table format? Different table formats require different amounts of memory to create the output HDFS file. Text and Sequencefiles need very little memory.
Hive copies data from table2 to a local files on some of the servers, then copies…"
I am trying to understand how does hive insert into with select works at memory and storage level.One of my oozie workflow fails due to GC overhead issue.Our query is: insert into table <> select a,b,c,d from table2.Now the map reduce job finished and then we get the message :Ended job = jobid: <>Loading data to table table2 Failed with exception java.lang.outofmemoryError: GC overhead limit exceeded.Failed: execution error, return code 1 from…See More
Graduate student in Big Data Systems. Currently working on transforming relational Database, Minibase to a Graph database. Enthusiastic of Data Mining advancements, new predictive models for data classification and extractions.