hadoop - Apache hive MSCK REPAIR TABLE new partition not added -


i new apache hive. while working on external table partition, if add new partition directly hdfs, new partition not added after running msck repair table. below codes tried,

-- creating external table

hive> create external table factory(name string, empid int, age int) partitioned by(region string)       > row format delimited fields terminated ',';  

--detailed table information

location:  hdfs://localhost.localdomain:8020/user/hive/warehouse/factory      table type:             external_table            table parameters:             external                true                     transient_lastddltime   1438579844   

-- creating directory in hdfs load data table factory

[cloudera@localhost ~]$ hadoop fs -mkdir 'hdfs://localhost.localdomain:8020/user/hive/testing/testing1/factory1' [cloudera@localhost ~]$ hadoop fs -mkdir 'hdfs://localhost.localdomain:8020/user/hive/testing/testing1/factory2' 

-- table data

cat factory1.txt emp1,500,40 emp2,501,45 emp3,502,50  cat factory2.txt emp10,200,25 emp11,201,27 emp12,202,30 

-- copying local hdfs

[cloudera@localhost ~]$ hadoop fs -copyfromlocal '/home/cloudera/factory1.txt' 'hdfs://localhost.localdomain:8020/user/hive/testing/testing1/factory1' [cloudera@localhost ~]$ hadoop fs -copyfromlocal '/home/cloudera/factory2.txt' 'hdfs://localhost.localdomain:8020/user/hive/testing/testing1/factory2' 

-- altering table update in metastore

hive> alter table factory add partition(region='southregion') location '/user/hive/testing/testing1/factory2'; hive> alter table factory add partition(region='northregion') location '/user/hive/testing/testing1/factory1';             hive> select * factory;                                                                       ok emp1    500 40  northregion emp2    501 45  northregion emp3    502 50  northregion emp10   200 25  southregion emp11   201 27  southregion emp12   202 30  southregion 

now created new file factory3.txt add new partition table factory

cat factory3.txt user1,100,25 user2,101,27 user3,102,30 

-- creating path , copying table data

[cloudera@localhost ~]$ hadoop fs -mkdir 'hdfs://localhost.localdomain:8020/user/hive/testing/testing1/factory2' [cloudera@localhost ~]$ hadoop fs -copyfromlocal '/home/cloudera/factory3.txt' 'hdfs://localhost.localdomain:8020/user/hive/testing/testing1/factory3' 

now executed below query update metastore new partition added

msck repair table factory; 

now table not giving new partition content of factory3 file. can know doing mistake while adding partition table factory?

whereas, if run alter command showing new partition data.

hive> alter table factory add partition(region='eastregion') location '/user/hive/testing/testing1/factory3'; 

can know why msck repair table command not working?

for msck work, naming convention /partition_name=partition_value/ should used.


Comments

Popular posts from this blog

c - Bitwise operation with (signed) enum value -

xslt - Unnest parent nodes by child node -

YouTubePlayerFragment cannot be cast to android.support.v4.app.Fragment -