hadoop - Apache hive MSCK REPAIR TABLE new partition not added -
i new apache hive. while working on external table partition, if add new partition directly hdfs, new partition not added after running msck repair table. below codes tried,
-- creating external table
hive> create external table factory(name string, empid int, age int) partitioned by(region string) > row format delimited fields terminated ',';
--detailed table information
location: hdfs://localhost.localdomain:8020/user/hive/warehouse/factory table type: external_table table parameters: external true transient_lastddltime 1438579844
-- creating directory in hdfs load data table factory
[cloudera@localhost ~]$ hadoop fs -mkdir 'hdfs://localhost.localdomain:8020/user/hive/testing/testing1/factory1' [cloudera@localhost ~]$ hadoop fs -mkdir 'hdfs://localhost.localdomain:8020/user/hive/testing/testing1/factory2'
-- table data
cat factory1.txt emp1,500,40 emp2,501,45 emp3,502,50 cat factory2.txt emp10,200,25 emp11,201,27 emp12,202,30
-- copying local hdfs
[cloudera@localhost ~]$ hadoop fs -copyfromlocal '/home/cloudera/factory1.txt' 'hdfs://localhost.localdomain:8020/user/hive/testing/testing1/factory1' [cloudera@localhost ~]$ hadoop fs -copyfromlocal '/home/cloudera/factory2.txt' 'hdfs://localhost.localdomain:8020/user/hive/testing/testing1/factory2'
-- altering table update in metastore
hive> alter table factory add partition(region='southregion') location '/user/hive/testing/testing1/factory2'; hive> alter table factory add partition(region='northregion') location '/user/hive/testing/testing1/factory1'; hive> select * factory; ok emp1 500 40 northregion emp2 501 45 northregion emp3 502 50 northregion emp10 200 25 southregion emp11 201 27 southregion emp12 202 30 southregion
now created new file factory3.txt add new partition table factory
cat factory3.txt user1,100,25 user2,101,27 user3,102,30
-- creating path , copying table data
[cloudera@localhost ~]$ hadoop fs -mkdir 'hdfs://localhost.localdomain:8020/user/hive/testing/testing1/factory2' [cloudera@localhost ~]$ hadoop fs -copyfromlocal '/home/cloudera/factory3.txt' 'hdfs://localhost.localdomain:8020/user/hive/testing/testing1/factory3'
now executed below query update metastore new partition added
msck repair table factory;
now table not giving new partition content of factory3 file. can know doing mistake while adding partition table factory?
whereas, if run alter command showing new partition data.
hive> alter table factory add partition(region='eastregion') location '/user/hive/testing/testing1/factory3';
can know why msck repair table command not working?
for msck work, naming convention /partition_name=partition_value/ should used.
Comments
Post a Comment