关于mysql的错误 - no query specified

本文介绍了在使用MySQL时遇到的错误:noqueryspecified的原因及解决方法。通常此错误发生在SQL语句末尾多输入了分号的情况下,例如使用G命令后误加了分号。了解这些常见错误有助于提高SQL查询效率。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

Mysql----error:no query specified

mysql下抛出错误:
error:no query specified
出现此错误是sql不合法原因:
如:select * from abc\G;
或者
select * from abc;;

\G后面不需要再加分号;


[root@node ~]# start-dfs.sh Starting namenodes on [node] Last login: 二 7月 8 16:00:18 CST 2025 from 192.168.1.92 on pts/0 Starting datanodes Last login: 二 7月 8 16:00:38 CST 2025 on pts/0 Starting secondary namenodes [node] Last login: 二 7月 8 16:00:41 CST 2025 on pts/0 SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [root@node ~]# start-yarn.sh Starting resourcemanager Last login: 二 7月 8 16:00:45 CST 2025 on pts/0 Starting nodemanagers Last login: 二 7月 8 16:00:51 CST 2025 on pts/0 [root@node ~]# mapred --daemon start historyserver [root@node ~]# jps 3541 ResourceManager 4007 Jps 2984 NameNode 3944 JobHistoryServer 3274 SecondaryNameNode [root@node ~]# mkdir -p /weblog [root@node ~]# cat > /weblog/access.log << EOF > 192.168.1.1,2023-06-01 10:30:22,/index.html > 192.168.1.2,2023-06-01 10:31:15,/product.html > 192.168.1.1,2023-06-01 10:32:45,/cart.html > 192.168.1.3,2023-06-01 11:45:30,/checkout.html > 192.168.1.4,2023-06-01 12:10:05,/index.html > 192.168.1.2,2023-06-01 14:20:18,/product.htm > EOF [root@node ~]# ls /weblog access.log [root@node ~]# hdfs dfs -mkdir -p /weblog/raw SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [root@node ~]# hdfs dfs -put /weblog/access.log /weblog/raw/ SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [root@node ~]# hdfs dfs -ls /weblog/raw SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Found 1 items -rw-r--r-- 3 root supergroup 269 2025-07-08 16:03 /weblog/raw/access.log [root@node ~]# cd /weblog [root@node weblog]# mkdir weblog-mapreduce [root@node weblog]# cd weblog-mapreduce [root@node weblog-mapreduce]# touch CleanMapper.java [root@node weblog-mapreduce]# vim CleanMapper.java import java.io.IOException; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; public class CleanMapper extends Mapper<LongWritable, Text, Text, NullWritable> { public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); String[] fields = line.split(","); if(fields.length == 3) { String ip = fields[0]; String time = fields[1]; String page = fields[2]; if(ip.matches("\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}")) { String outputLine = ip + "," + time + "," + page; context.write(new Text(outputLine), NullWritable.get()); } } } } [root@node weblog-mapreduce]# touch CleanReducer.java [root@node weblog-mapreduce]# vim CleanReducer.java import java.io.IOException; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; public class CleanReducer extends Reducer<Text, NullWritable, Text, NullWritable> { public void reduce(Text key, Iterable<NullWritable> values, Context context) throws IOException, InterruptedException { context.write(key, NullWritable.get()); } } [root@node weblog-mapreduce]# touch LogCleanDriver.java [root@node weblog-mapreduce]# vim LogCleanDriver.java import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class LogCleanDriver { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "Web Log Cleaner"); job.setJarByClass(LogCleanDriver.class); job.setMapperClass(CleanMapper.class); job.setReducerClass(CleanReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(NullWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } } [root@node weblog-mapreduce]# ls /weblog/weblog-mapreduce CleanMapper.java CleanReducer.java LogCleanDriver.java [root@node weblog-mapreduce]# javac -classpath $(hadoop classpath) -d . *.java [root@node weblog-mapreduce]# ls /weblog/weblog-mapreduce CleanMapper.class CleanReducer.class LogCleanDriver.class CleanMapper.java CleanReducer.java LogCleanDriver.java [root@node weblog-mapreduce]# jar cf logclean.jar *.class [root@node weblog-mapreduce]# ls /weblog/weblog-mapreduce CleanMapper.class CleanReducer.class LogCleanDriver.class logclean.jar CleanMapper.java CleanReducer.java LogCleanDriver.java [root@node weblog-mapreduce]# hdfs dfs -ls /weblog/raw SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Found 1 items -rw-r--r-- 3 root supergroup 269 2025-07-08 16:03 /weblog/raw/access.log [root@node weblog-mapreduce]# hdfs dfs -ls /weblog/output SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. ls: `/weblog/output': No such file or directory [root@node weblog-mapreduce]# hadoop jar logclean.jar LogCleanDriver /weblog/raw /weblog/output SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [root@node weblog-mapreduce]# [root@node weblog-mapreduce]# mapred job -status job_1751961655287_0001 SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Job: job_1751961655287_0001 Job File: hdfs://node:9000/tmp/hadoop-yarn/staging/history/done/2025/07/08/000000/job_1751961655287_0001_conf.xml Job Tracking URL : http://node:19888/jobhistory/job/job_1751961655287_0001 Uber job : false Number of maps: 1 Number of reduces: 1 map() completion: 1.0 reduce() completion: 1.0 Job state: SUCCEEDED retired: false reason for failure: Counters: 54 File System Counters FILE: Number of bytes read=287 FILE: Number of bytes written=552699 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=372 HDFS: Number of bytes written=269 HDFS: Number of read operations=8 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 HDFS: Number of bytes read erasure-coded=0 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=1848 Total time spent by all reduces in occupied slots (ms)=2016 Total time spent by all map tasks (ms)=1848 Total time spent by all reduce tasks (ms)=2016 Total vcore-milliseconds taken by all map tasks=1848 Total vcore-milliseconds taken by all reduce tasks=2016 Total megabyte-milliseconds taken by all map tasks=1892352 Total megabyte-milliseconds taken by all reduce tasks=2064384 Map-Reduce Framework Map input records=6 Map output records=6 Map output bytes=269 Map output materialized bytes=287 Input split bytes=103 Combine input records=0 Combine output records=0 Reduce input groups=6 Reduce shuffle bytes=287 Reduce input records=6 Reduce output records=6 Spilled Records=12 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=95 CPU time spent (ms)=1050 Physical memory (bytes) snapshot=500764672 Virtual memory (bytes) snapshot=5614292992 Total committed heap usage (bytes)=379584512 Peak Map Physical memory (bytes)=293011456 Peak Map Virtual memory (bytes)=2803433472 Peak Reduce Physical memory (bytes)=207753216 Peak Reduce Virtual memory (bytes)=2810859520 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=269 File Output Format Counters Bytes Written=269 [root@node weblog-mapreduce]# hdfs dfs -ls /weblog/output SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Found 2 items -rw-r--r-- 3 root supergroup 0 2025-07-08 16:34 /weblog/output/_SUCCESS -rw-r--r-- 3 root supergroup 269 2025-07-08 16:34 /weblog/output/part-r-00000 [root@node weblog-mapreduce]# hdfs dfs -cat /weblog/output/part-r-00000 | head -5 SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. 192.168.1.1,2023-06-01 10:30:22,/index.html 192.168.1.1,2023-06-01 10:32:45,/cart.html 192.168.1.2,2023-06-01 10:31:15,/product.html 192.168.1.2,2023-06-01 14:20:18,/product.htm 192.168.1.3,2023-06-01 11:45:30,/checkout.html [root@node weblog-mapreduce]# hive SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Hive Session ID = 5199f37c-a381-428a-be1b-0a2afaab8583 Logging initialized using configuration in jar:file:/home/hive-3.1.3/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Hive Session ID = f38c99b3-ff7c-4f61-ae07-6b21d86d7160 hive> CREATE EXTERNAL TABLE weblog ( > ip STRING, > access_time TIMESTAMP, > page STRING > ) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY ',' > LOCATION '/weblog/output'; OK Time taken: 1.274 seconds hive> select * from weblog; OK 192.168.1.1 2023-06-01 10:30:22 /index.html 192.168.1.1 2023-06-01 10:32:45 /cart.html 192.168.1.2 2023-06-01 10:31:15 /product.html 192.168.1.2 2023-06-01 14:20:18 /product.htm 192.168.1.3 2023-06-01 11:45:30 /checkout.html 192.168.1.4 2023-06-01 12:10:05 /index.html Time taken: 1.947 seconds, Fetched: 6 row(s) hive> select * from weblog limit 5; OK 192.168.1.1 2023-06-01 10:30:22 /index.html 192.168.1.1 2023-06-01 10:32:45 /cart.html 192.168.1.2 2023-06-01 10:31:15 /product.html 192.168.1.2 2023-06-01 14:20:18 /product.htm 192.168.1.3 2023-06-01 11:45:30 /checkout.html Time taken: 0.148 seconds, Fetched: 5 row(s) hive> hive> CREATE TABLE page_visits AS > SELECT > page, > COUNT(*) AS visits > FROM weblog > GROUP BY page > ORDER BY visits DESC; Query ID = root_20250708183002_ec44d1b4-af24-403c-bb67-380dfb6961c3 Total jobs = 2 Launching Job 1 out of 2 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1751961655287_0002, Tracking URL = http://node:8088/proxy/application_1751961655287_0002/ Kill Command = /home/hadoop/hadoop3.3/bin/mapred job -kill job_1751961655287_0002 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2025-07-08 18:30:12,692 Stage-1 map = 0%, reduce = 0% 2025-07-08 18:30:16,978 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.8 sec 2025-07-08 18:30:23,184 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.66 sec MapReduce Total cumulative CPU time: 3 seconds 660 msec Ended Job = job_1751961655287_0002 Launching Job 2 out of 2 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1751961655287_0003, Tracking URL = http://node:8088/proxy/application_1751961655287_0003/ Kill Command = /home/hadoop/hadoop3.3/bin/mapred job -kill job_1751961655287_0003 Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1 2025-07-08 18:30:35,969 Stage-2 map = 0%, reduce = 0% 2025-07-08 18:30:41,155 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.23 sec 2025-07-08 18:30:46,313 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.95 sec MapReduce Total cumulative CPU time: 2 seconds 950 msec Ended Job = job_1751961655287_0003 Moving data to directory hdfs://node:9000/hive/warehouse/page_visits MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 3.66 sec HDFS Read: 12379 HDFS Write: 251 SUCCESS Stage-Stage-2: Map: 1 Reduce: 1 Cumulative CPU: 2.95 sec HDFS Read: 7308 HDFS Write: 150 SUCCESS Total MapReduce CPU Time Spent: 6 seconds 610 msec OK Time taken: 46.853 seconds hive> hive> describe page_visits; OK page string visits bigint Time taken: 0.214 seconds, Fetched: 2 row(s) hive> CREATE TABLE ip_visits AS > SELECT > ip, > COUNT(*) AS visits > FROM weblog > GROUP BY ip > ORDER BY visits DESC; Query ID = root_20250708183554_da402d08-af34-46f9-a33a-3f66ddd1a580 Total jobs = 2 Launching Job 1 out of 2 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1751961655287_0004, Tracking URL = http://node:8088/proxy/application_1751961655287_0004/ Kill Command = /home/hadoop/hadoop3.3/bin/mapred job -kill job_1751961655287_0004 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2025-07-08 18:36:04,037 Stage-1 map = 0%, reduce = 0% 2025-07-08 18:36:09,250 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.57 sec 2025-07-08 18:36:14,393 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.3 sec MapReduce Total cumulative CPU time: 3 seconds 300 msec Ended Job = job_1751961655287_0004 Launching Job 2 out of 2 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1751961655287_0005, Tracking URL = http://node:8088/proxy/application_1751961655287_0005/ Kill Command = /home/hadoop/hadoop3.3/bin/mapred job -kill job_1751961655287_0005 Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1 2025-07-08 18:36:27,073 Stage-2 map = 0%, reduce = 0% 2025-07-08 18:36:31,215 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.25 sec 2025-07-08 18:36:36,853 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.27 sec MapReduce Total cumulative CPU time: 3 seconds 270 msec Ended Job = job_1751961655287_0005 Moving data to directory hdfs://node:9000/hive/warehouse/ip_visits MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 3.3 sec HDFS Read: 12445 HDFS Write: 216 SUCCESS Stage-Stage-2: Map: 1 Reduce: 1 Cumulative CPU: 3.27 sec HDFS Read: 7261 HDFS Write: 129 SUCCESS Total MapReduce CPU Time Spent: 6 seconds 570 msec OK Time taken: 44.523 seconds hive> [root@node weblog-mapreduce]# hive> [root@node weblog-mapreduce]# describe ip_visite; bash: describe: command not found... [root@node weblog-mapreduce]# hive SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Hive Session ID = 57dafc2a-afe2-41a4-8159-00f8d44b5add Logging initialized using configuration in jar:file:/home/hive-3.1.3/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true Hive Session ID = f866eae4-4cb4-4403-b7a2-7a52701c5a74 Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. hive> describe ip_visite; FAILED: SemanticException [Error 10001]: Table not found ip_visite hive> describe ip_visits; OK ip string visits bigint Time taken: 0.464 seconds, Fetched: 2 row(s) hive> SELECT * FROM page_visits; OK /index.html 2 /product.html 1 /product.htm 1 /checkout.html 1 /cart.html 1 Time taken: 2.095 seconds, Fetched: 5 row(s) hive> SELECT * FROM ip_visits; OK 192.168.1.2 2 192.168.1.1 2 192.168.1.4 1 192.168.1.3 1 Time taken: 0.176 seconds, Fetched: 4 row(s) hive> hive> [root@node weblog-mapreduce]# [root@node weblog-mapreduce]# mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 48 Server version: 8.0.42 MySQL Community Server - GPL Copyright (c) 2000, 2025, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> CREATE DATABASE IF NOT EXISTS weblog_db; Query OK, 1 row affected (0.06 sec) mysql> USE weblog_db; Database changed mysql> CREATE TABLE IF NOT EXISTS page_visits ( -> page VARCHAR(255), -> visits BIGINT -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8; Query OK, 0 rows affected, 1 warning (0.05 sec) mysql> SHOW TABLES; +---------------------+ | Tables_in_weblog_db | +---------------------+ | page_visits | +---------------------+ 1 row in set (0.00 sec) mysql> DESCRIBE page_visits; +--------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------+--------------+------+-----+---------+-------+ | page | varchar(255) | YES | | NULL | | | visits | bigint | YES | | NULL | | +--------+--------------+------+-----+---------+-------+ 2 rows in set (0.00 sec) mysql> CREATE TABLE IF NOT EXISTS ip_visits ( -> ip VARCHAR(15), -> visits BIGINT -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8; Query OK, 0 rows affected, 1 warning (0.02 sec) mysql> SHOW TABLES; +---------------------+ | Tables_in_weblog_db | +---------------------+ | ip_visits | | page_visits | +---------------------+ 2 rows in set (0.01 sec) mysql> DESC ip_visits; +--------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------+-------------+------+-----+---------+-------+ | ip | varchar(15) | YES | | NULL | | | visits | bigint | YES | | NULL | | +--------+-------------+------+-----+---------+-------+ 2 rows in set (0.00 sec) mysql> ^C mysql> [root@node weblog-mapreduce]# hive SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Hive Session ID = f34e6971-71ae-4aa5-aa22-895061f33bdf Logging initialized using configuration in jar:file:/home/hive-3.1.3/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Hive Session ID = f7a06e76-e117-4fbb-9ee8-09fdfd002104 hive> DESCRIBE FORMATTED page_visits; OK # col_name data_type comment page string visits bigint # Detailed Table Information Database: default OwnerType: USER Owner: root CreateTime: Tue Jul 08 18:30:47 CST 2025 LastAccessTime: UNKNOWN Retention: 0 Location: hdfs://node:9000/hive/warehouse/page_visits Table Type: MANAGED_TABLE Table Parameters: COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"} bucketing_version 2 numFiles 1 numRows 5 rawDataSize 70 totalSize 75 transient_lastDdlTime 1751970648 # Storage Information SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe InputFormat: org.apache.hadoop.mapred.TextInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Compressed: No Num Buckets: -1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: serialization.format 1 Time taken: 1.043 seconds, Fetched: 32 row(s) hive> 到这里就不会了 6.2.2sqoop导出格式 6.2.3导出page_visits表 6.2.4导出到ip_visits表 6.3验证导出数据 6.3.1登录MySQL 6.3.2执行查询
最新发布
07-09
<think>我们当前的任务是使用Sqoop将Hive中的表(page_visits和ip_visits)导出到MySQL数据库中。在此之前,我们已经创建了MySQL数据库weblog_db和相应的表(page_visits和ip_visits)。 步骤: 1. 确认Sqoop已经安装并配置正确(包括MySQL JDBC驱动)。 2. 使用Sqoop的export命令将Hive表中的数据导出到MySQL的对应表中。 注意:Hive中的表是存储在HDFS上的,我们可以通过指定Hive表的HDFS路径或者直接使用Hive表名(但Sqoop需要知道HDFS路径)来导出。 在Hive中,我们之前创建的表page_visits和ip_visits都是内部表(MANAGED_TABLE),其数据存储在HDFS上。通过`DESCRIBE FORMATTED page_visits;`我们可以获取到表的HDFS路径(Location字段)。但是,Sqoop可以直接连接Hive元数据获取表的信息,也可以直接指定HDFS路径。 这里我们选择直接使用Hive表名(通过`--hcatalog-table`参数)或者使用HDFS路径(通过`--export-dir`参数)。但是,由于我们使用的是Hive内部表,我们可以使用`--export-dir`参数指定HDFS路径。 然而,在Sqoop export中,我们通常使用`--export-dir`参数指定HDFS上的数据路径,同时指定目标MySQL表。 但是,注意:Hive表的数据格式可能是文本格式(我们创建的是文本格式),而MySQL表是关系型表。Sqoop默认可以处理文本格式的导出。 具体步骤: 6.2.2 Sqoop导出格式 Sqoop导出支持多种格式,包括文本文件、SequenceFile、Avro等。由于我们的Hive表是文本格式,所以使用默认的文本格式即可。 6.2.3 导出page_visits表 首先,我们需要知道Hive表page_visits在HDFS上的路径。根据之前的Hive命令输出,page_visits表的Location是:`hdfs://node:9000/hive/warehouse/page_visits` 然后,使用Sqoop export命令将数据导出到MySQL的page_visits表中。 6.2.4 导出ip_visits表 同理,ip_visits表的Location可以通过类似方式获得,或者我们也可以使用Hive的`DESCRIBE FORMATTED ip_visits;`来查看。但根据之前创建表的逻辑,它应该位于`hdfs://node:9000/hive/warehouse/ip_visits`。 6.3 验证导出数据 登录MySQL,执行查询。 具体命令: 1. 导出page_visits表到MySQL: ```bash sqoop export \ --connect jdbc:mysql://node:3306/weblog_db \ --username root \ --password your_mysql_password \ --table page_visits \ --export-dir /hive/warehouse/page_visits \ --input-fields-terminated-by '\001' # 注意:Hive默认的字段分隔符是'\001'(Ctrl+A),但我们在创建表时指定了逗号分隔?实际上,我们创建表时指定了逗号分隔,但Hive默认存储时使用的是\001。然而,我们在创建表时指定了ROW FORMAT DELIMITED FIELDS TERMINATED BY ',',所以数据应该是逗号分隔的。因此,这里应该使用逗号分隔符。 ``` 但是,我们如何知道Hive表的数据分隔符?通过之前创建表的语句,我们指定了逗号分隔。所以,在导出时应该指定分隔符为逗号。 修正:在创建Hive表weblog时,我们指定了逗号分隔。但是,我们后来通过CTAS(CREATE TABLE AS SELECT)创建了page_visits和ip_visits,这两个表没有指定分隔符,所以它们会使用Hive默认的分隔符,即\001。 因此,我们需要确认这两个表的分隔符。通过`DESCRIBE FORMATTED`命令可以查看。 在之前的Hive会话中,我们执行了`DESCRIBE FORMATTED page_visits;`,在输出中,我们注意到: ``` Storage Desc Params: serialization.format 1 ``` 这里并没有明确的分隔符信息,但Hive内部表默认使用\001作为分隔符。 所以,为了保险,我们可以使用`--input-fields-terminated-by '\001'`。但是,我们也可以查看HDFS上的数据文件来确定分隔符。 不过,根据我们之前创建Hive表的方式(CTAS),并且没有指定分隔符,所以是默认的\001。 因此,导出命令应该指定分隔符为\001。 但是,如果我们不确定,可以查看HDFS上的文件内容: ```bash hdfs dfs -cat /hive/warehouse/page_visits/000000_0 ``` 如果显示的是不可见字符,那么很可能是\001。如果看到的是逗号分隔,那么就是逗号。 根据之前的Hive查询结果,page_visits表的内容是: /index.html 2 /product.html 1 ... 在Hive中显示时是用制表符分隔的?实际上,Hive的显示格式和存储格式是分开的。存储格式是使用我们创建表时指定的分隔符,而CTAS创建的表没有指定,所以是默认的\001。 所以,我们导出时应该使用`--input-fields-terminated-by '\001'`。 但是,在Sqoop中,\001的表示可以是`--input-fields-terminated-by '\001'`,也可以使用`--input-fields-terminated-by "\001"`(双引号),或者使用`--input-fields-terminated-by \\001`。不过,在命令行中,我们通常用单引号括起来。 另外,我们还需要注意MySQL中的表结构和Hive表结构(字段顺序和类型)要一致。 开始导出: 1. 导出page_visits表: ```bash sqoop export \ --connect "jdbc:mysql://node:3306/weblog_db?useUnicode=true&characterEncoding=utf-8" \ --username root \ --password your_password \ --table page_visits \ --export-dir /hive/warehouse/page_visits \ --input-fields-terminated-by '\001' ``` 2. 导出ip_visits表: ```bash sqoop export \ --connect "jdbc:mysql://node:3306/weblog_db?useUnicode=true&characterEncoding=utf-8" \ --username root \ --password your_password \ --table ip_visits \ --export-dir /hive/warehouse/ip_visits \ --input-fields-terminated-by '\001' ``` 注意:将your_password替换为实际的MySQL root密码。 6.3 验证导出数据 登录MySQL,执行查询: ```sql USE weblog_db; SELECT * FROM page_visits; SELECT * FROM ip_visits; ``` 如果数据一致,则导出成功。 但是,我们可能会遇到错误,比如分隔符不对,或者字段数量不匹配等。所以,在导出之前,最好确认Hive表的数据结构和MySQL表的结构完全一致(字段顺序、类型、分隔符等)。 另外,如果Hive表中有NULL值,我们可能需要指定MySQL如何处理NULL值(例如,通过`--input-null-string`和`--input-null-non-string`参数)。 由于我们的数据中可能没有NULL值,所以暂时不指定。 现在,我们按照上述步骤执行。 注意:在Sqoop命令中,我们使用了`--export-dir`参数,指定的是Hive表在HDFS上的路径。这个路径可以通过Hive的`DESCRIBE FORMATTED table_name`命令的Location字段获得。 在之前的Hive查询中,page_visits的Location是`hdfs://node:9000/hive/warehouse/page_visits`,但在Sqoop中,我们可以省略hdfs://node:9000,因为Sqoop会使用默认的Hadoop配置中的文件系统地址。所以,我们只需要写`/hive/warehouse/page_visits`。 现在,我们执行Sqoop导出命令。 由于在命令行中直接写密码不安全,可以使用-P参数,这样Sqoop会提示输入密码。 或者,将密码保存在文件中(使用--password-file参数),但这里为了简单,我们使用-P。 但是,在脚本中,我们通常使用--password参数,但这样会暴露密码。所以,在实验环境中,我们可以直接使用。 开始执行: 首先,导出page_visits表: ```bash sqoop export \ --connect "jdbc:mysql://node:3306/weblog_db" \ --username root \ --password your_mysql_root_password \ --table page_visits \ --export-dir /hive/warehouse/page_visits \ --input-fields-terminated-by '\001' ``` 然后,导出ip_visits表: ```bash sqoop export \ --connect "jdbc:mysql://node:3306/weblog_db" \ --username root \ --password your_mysql_root_password \ --table ip_visits \ --export-dir /hive/warehouse/ip_visits \ --input-fields-terminated-by '\001' ``` 最后,登录MySQL验证。 如果遇到错误,比如: - 连接问题:检查MySQL服务是否运行,以及是否允许远程连接(这里我们是本地连接,所以应该没问题)。 - 表不存在:检查MySQL中的表名是否正确。 - 列数不匹配:检查Hive表和MySQL表的列数是否一致。 在导出过程中,Sqoop会启动MapReduce任务,我们可以通过YARN的Web UI查看任务状态。 现在,我们给出完整的步骤。</think>### 6.2.2 Sqoop导出格式说明 Sqoop支持多种导出格式,最常用的是: 1. **文本格式**:默认格式,使用逗号、制表符等分隔符 2. **Avro格式**:二进制格式,保留数据类型信息 3. **SequenceFile格式**:Hadoop原生二进制格式 4. **Parquet格式**:列式存储格式 对于关系型数据库导出,文本格式是最简单直接的选择,我们将使用默认的文本格式导出。 ### 6.2.3 导出page_visits表到MySQL 使用Sqoop将Hive中的`page_visits`表导出到MySQL: ```bash # 导出page_visits表 sqoop export \ --connect jdbc:mysql://localhost:3306/weblog_db \ --username root \ --password your_mysql_password \ --table page_visits \ --export-dir /hive/warehouse/page_visits \ --input-fields-terminated-by '\001' \ --input-lines-terminated-by '\n' ``` **参数解释:** - `--connect`:MySQL数据库连接URL - `--username`:MySQL用户名 - `--password`:MySQL密码(替换为实际密码) - `--table`:目标MySQL表名 - `--export-dir`:Hive表在HDFS上的存储路径 - `--input-fields-terminated-by`:指定字段分隔符(Hive默认使用\001) - `--input-lines-terminated-by`:指定行分隔符(默认为换行符) ### 6.2.4 导出ip_visits表到MySQL 使用Sqoop将Hive中的`ip_visits`表导出到MySQL: ```bash # 导出ip_visits表 sqoop export \ --connect jdbc:mysql://localhost:3306/weblog_db \ --username root \ --password your_mysql_password \ --table ip_visits \ --export-dir /hive/warehouse/ip_visits \ --input-fields-terminated-by '\001' \ --input-lines-terminated-by '\n' ``` ### 6.3 验证导出数据 #### 6.3.1 登录MySQL ```bash mysql -u root -p ``` #### 6.3.2 执行查询验证数据 ```sql USE weblog_db; -- 验证page_visits表 SELECT * FROM page_visits; -- 验证ip_visits表 SELECT * FROM ip_visits; -- 统计总记录数 SELECT (SELECT COUNT(*) FROM page_visits) AS page_visits_count, (SELECT COUNT(*) FROM ip_visits) AS ip_visits_count; ``` **预期输出:** ``` +----------------+----------------+ | page_visits_count | ip_visits_count | +----------------+----------------+ | 5 | 4 | +----------------+----------------+ ``` ### 完整流程说明 1. **Sqoop导出原理**: - Sqoop启动MapReduce作业 - Mapper读取HDFS上的数据文件 - 将数据转换为SQL INSERT语句 - 批量提交到MySQL数据库 2. **分隔符说明**: - Hive内部表默认使用`\001`(Ctrl-A)作为字段分隔符 - 使用`\n`作为行分隔符 - 必须正确指定分隔符才能确保数据正确解析 3. **数据一致性检查**: - 比较Hive表和MySQL表的记录数 - 随机抽样检查数据内容是否一致 - 验证特殊字符是否正确处理 ### 常见问题解决 1. **导出失败:权限问题** ```bash # 在MySQL中授权 mysql> GRANT ALL PRIVILEGES ON weblog_db.* TO 'root'@'localhost'; ``` 2. **导出失败:表不存在** ```sql -- 确认MySQL表存在 USE weblog_db; SHOW TABLES; ``` 3. **数据乱码:字符集问题** ```bash # 添加字符集参数 --connect "jdbc:mysql://localhost:3306/weblog_db?useUnicode=true&characterEncoding=UTF-8" ``` 4. **数据截断:字段长度不足** ```sql -- 调整MySQL表结构 ALTER TABLE page_visits MODIFY page VARCHAR(500); ``` ###
评论 6
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值