注册登录
提 问
English中文版
软件开发数据库Hadoop数据抽取hivesqoop1invalid path

Sqoop1从DB2中抽数据到Hive出现异常

使用Sqoop将数据库中的一个表抽取到Hive中，运行如下命令
sqoop import --connect jdbc:db2://10.158.130.46:50000/hndcdb --username DWINST --password 'hndw&83d' --table DCDW.FACT_FI_INDUS_FIN_QUICK_REPORT_T --fields-terminated-by "," --lines-terminated-by "n" --hive-import --create-hive-table --hive-overwrite --hive-table wang.fact_fi_indus_fin_quick_report_t -m1
运行最后报错：
15/03/05 15:10:30 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/03/05 15:10:30 INFO manager.SqlManager: Using default fetchSize of 1000
15/03/05 15:10:30 INFO tool.CodeGenTool: Beginning code generation
15/03/05 15:10:30 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM DCDW.FACT_FI_INDUS_FIN_QUICK_REPORT_T AS t WHERE 1=0
15/03/05 15:10:30 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM DCDW.FACT_FI_INDUS_FIN_QUICK_REPORT_T AS t WHERE 1=0
15/03/05 15:10:30 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
15/03/05 15:10:30 INFO orm.CompilationManager: Found hadoop core jar at: /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar
注: /tmp/sqoop-dctest/compile/ea37080d60e20496a0746bb154ebfd5b/DCDW_FACT_FI_INDUS_FIN_QUICK_REPORT_T.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
15/03/05 15:10:32 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-dctest/compile/ea37080d60e20496a0746bb154ebfd5b/DCDW.FACT_FI_INDUS_FIN_QUICK_REPORT_T.jar
15/03/05 15:10:32 INFO mapreduce.ImportJobBase: Beginning import of DCDW.FACT_FI_INDUS_FIN_QUICK_REPORT_T
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/sqoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/03/05 15:10:32 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM DCDW.FACT_FI_INDUS_FIN_QUICK_REPORT_T AS t WHERE 1=0
15/03/05 15:10:33 INFO client.RMProxy: Connecting to ResourceManager at hdpmaster.hnzy.com/10.158.240.5:8032
15/03/05 15:10:35 INFO mapreduce.JobSubmitter: number of splits:1
15/03/05 15:10:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1423817137566_0354
15/03/05 15:10:35 INFO impl.YarnClientImpl: Submitted application application_1423817137566_0354
15/03/05 15:10:35 INFO mapreduce.Job: The url to track the job: http://hdpmaster.hnzy.com:8088/proxy/application_1423817137566_0354/
15/03/05 15:10:35 INFO mapreduce.Job: Running job: job_1423817137566_0354
15/03/05 15:10:42 INFO mapreduce.Job: Job job_1423817137566_0354 running in uber mode : false
15/03/05 15:10:42 INFO mapreduce.Job:  map 0% reduce 0%
15/03/05 15:10:50 INFO mapreduce.Job:  map 100% reduce 0%
15/03/05 15:10:50 INFO mapreduce.Job: Job job_1423817137566_0354 completed successfully
15/03/05 15:10:51 INFO mapreduce.Job: Counters: 30
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=137819
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=87
                HDFS: Number of bytes written=392456
                HDFS: Number of read operations=4
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=1
                Other local map tasks=1
                Total time spent by all maps in occupied slots (ms)=5463
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=5463
                Total vcore-seconds taken by all map tasks=5463
                Total megabyte-seconds taken by all map tasks=5594112
        Map-Reduce Framework
                Map input records=3942
                Map output records=3942
                Input split bytes=87
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=125
                CPU time spent (ms)=2420
                Physical memory (bytes) snapshot=206708736
                Virtual memory (bytes) snapshot=2697707520
                Total committed heap usage (bytes)=516489216
        File Input Format Counters 
                Bytes Read=0
        File Output Format Counters 
                Bytes Written=392456
15/03/05 15:10:51 INFO mapreduce.ImportJobBase: Transferred 383.2578 KB in 18.0517 seconds (21.2311 KB/sec)
15/03/05 15:10:51 INFO mapreduce.ImportJobBase: Retrieved 3942 records.
15/03/05 15:10:51 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM DCDW.FACT_FI_INDUS_FIN_QUICK_REPORT_T AS t WHERE 1=0
15/03/05 15:10:51 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM DCDW.FACT_FI_INDUS_FIN_QUICK_REPORT_T AS t WHERE 1=0
15/03/05 15:10:51 WARN hive.TableDefWriter: Column ID_KEY had to be cast to a less precise type in Hive
15/03/05 15:10:51 WARN hive.TableDefWriter: Column THIS_MONTH_QTY had to be cast to a less precise type in Hive
15/03/05 15:10:51 WARN hive.TableDefWriter: Column THIS_YEAR_QTY had to be cast to a less precise type in Hive
15/03/05 15:10:51 WARN hive.TableDefWriter: Column LAST_YEAR_QTY had to be cast to a less precise type in Hive
15/03/05 15:10:51 WARN hive.TableDefWriter: Column DT_ETLTIME had to be cast to a less precise type in Hive
15/03/05 15:10:51 INFO hive.HiveImport: Loading uploaded data into Hive
15/03/05 15:10:53 INFO hive.HiveImport: SLF4J: Class path contains multiple SLF4J bindings.
15/03/05 15:10:53 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/usr/lib/sqoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
15/03/05 15:10:53 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/usr/lib/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
15/03/05 15:10:53 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
15/03/05 15:10:53 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/usr/lib/ngmr/lib_managed/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
15/03/05 15:10:53 INFO hive.HiveImport: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
15/03/05 15:10:53 INFO hive.HiveImport: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/03/05 15:10:54 INFO hive.HiveImport: 15/03/05 15:10:54 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
15/03/05 15:10:54 INFO hive.HiveImport: 15/03/05 15:10:54 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
15/03/05 15:10:54 INFO hive.HiveImport: 15/03/05 15:10:54 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
15/03/05 15:10:54 INFO hive.HiveImport: 15/03/05 15:10:54 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
15/03/05 15:10:54 INFO hive.HiveImport: 15/03/05 15:10:54 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
15/03/05 15:10:54 INFO hive.HiveImport: 15/03/05 15:10:54 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
15/03/05 15:10:54 INFO hive.HiveImport: 15/03/05 15:10:54 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
15/03/05 15:10:58 INFO hive.HiveImport: OK
15/03/05 15:10:58 INFO hive.HiveImport: Time taken: 1.945 seconds
15/03/05 15:10:58 INFO hive.HiveImport: OK
15/03/05 15:10:58 INFO hive.HiveImport: Time taken: 0.266 seconds
15/03/05 15:10:59 INFO hive.HiveImport: FAILED: SemanticException Line 2:17 Invalid path ''hdfs://hdpmaster.hnzy.com:8020/user/dctest/DCDW.FACT_FI_INDUS_FIN_QUICK_REPORT_T''
15/03/05 15:10:59 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: Hive exited with status 64
        at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:364)
        at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:314)
        at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:226)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
本以为抽取失败，但是Hive中表已经建立好，不过没有数据，他所报的的Invalid path，中文件已经存在，直接使用
hive -e "load data inpaht "/user/dctest/DCDW.FACT_FI_INDUS_FIN_QUICK_REPORT_T" overwrite into table FACT_FI_INDUS_FIN_QUICK_REPORT_T"
可以将数据写进Hive表中，数据的记录条数与数据库也是一样。


123456.png



虽然可以分二步完成，但总是非常麻烦，希望大神指导一下。
关注1
评论
参与2

1同行回答
|
按赞同排序
按时间排序

张东焕张东焕技术总监IBM官方授权讲师
建议你看看这个http://www.ibm.com/developerwork ... ticle/dm-1212liuqy/显示全部
建议你看看这个

http://www.ibm.com/developerwork ... ticle/dm-1212liuqy/收起
IT咨询服务 · 2015-03-06
浏览917
评论

提问者

kyo19
kyo19
软件架构设计师whacko
关注
回答0
评论19

相关问题

  • 对于业务系统的历史数据存放、查询，哪种类型的数据库更为合适？
  • 超融合能否扛起全闪做支撑交易类数据库最佳平台的大旗？
  • 金融行业传统数据库一般如何解决io需求？
  • 信创环境下，哪些国产数据库能上银行核心系统？
  • 哪个国产数据库更值得去学习，各国产数据库的特点及缺点？

    • 相关资料

  • IDC+中国金融行业国产分布式数据库白皮书
  • Redhat7.6 Oracle 19C RAC部署手册
  • 《迈向YB数据时代》2023春季刊之——【趋势动态】议题1：大中型企业关键应用数据库未来会如何走向？
  • Redhat7.0 Oracle 12C RAC部署手册
  • 《迈向YB数据时代》2023春季刊之——【趋势动态】议题2：平滑替换架构未来是否会成为企业的主流架构？

    • 相关文章

  • Redis 的优势是如何在保单业务中发挥出来的
  • 南大通用对数据库领域国家标准编制与实施的贡献浅析
  • 数据库行业解决方案都写了啥
  • 企业传统Oracle数据库迁移国产数据库核心难点总结
  • 招聘：建信金科基础技术中心数据库产品团队

    • 问题状态

  • 发布时间：2015-03-05
  • 关注会员：1 人
  • 问题浏览：4180
  • 最近回答：2015-03-06

    • 关于TWT使用指南社区专家合作厂商入驻社区企业招聘投诉建议版权与免责声明联系我们
    © 2024talkwithtrend — talk with trend,talk with technologist京ICP备09031017号-30

    X社区推广