Spark入门（2）-Spark-Shell WordCount 单词统计

Spark-shell 是 Spark 给我们提供的交互式命令窗口，类似Linux的shell命令窗口。

启动spark

进入spark安装包，右键选择【在终端中打开】，使用的深度Linux系统。 输入启动命令：

./bin/spark-shell

可以通过IP：端口的形式查看spark web。

制作2个英文单词文件

我们就以spark启动的日志制作2个因为单词文件test1.txt于test2.txt。

Using Sparks default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
20/09/01 21:25:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/09/01 21:25:23 WARN Utils: Your hostname, *******-PC resolves to a loopback address: 127.0.1.1; using 192.168.31.194 instead (on interface wlp1s0)
20/09/01 21:25:23 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
20/09/01 21:25:29 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
20/09/01 21:25:29 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
20/09/01 21:25:30 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.31.194:4040
Spark context available as sc (master = local[*], app id = local-1598966724416).
Spark session available as spark.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _ / _ / _ `/ __/  _/
   /___/ .__/\_,_/_/ /_/\_   version 2.1.1
      /_/
         
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_261)
Type in expressions to have them evaluated.
Type :help for more information.

运行spark统计单词

sc.textFile("input/").flatMap(_.split("\W+")).map((_, 1)).reduceByKey(_ + _).collect

查看spark-jobs

命令解释

1. textFile(“input”)：读取本地文件input文件夹数据；
2. flatMap(_.split(" "))：压平操作，按照空格分割符将一行数据映射成一个个单词；
3. map((_,1))：对每一个元素操作，将单词映射为元组；
4. reduceByKey(+)：按照key将值进行聚合，相加；
5. collect：将数据收集到Driver端展示。