日本免费全黄少妇一区二区三区-高清无码一区二区三区四区-欧美中文字幕日韩在线观看-国产福利诱惑在线网站-国产中文字幕一区在线-亚洲欧美精品日韩一区-久久国产精品国产精品国产-国产精久久久久久一区二区三区-欧美亚洲国产精品久久久久

  1. 首頁 > 云知道 > >

Spark使用OSS Select加速數(shù)據(jù)查詢( 二 )

云知道


-rw-r--r-- root/root 67758 2018-10-30 16:11 spark-2.2.0-oss-select-0.1.0-SNAPSHOT/jettison-1.1.jar
-rw-r--r-- root/root 57264 2018-10-30 16:11 spark-2.2.0-oss-select-0.1.0-SNAPSHOT/json-20170516.jar
-rw-r--r-- root/root 890168 2018-10-30 16:11 spark-2.2.0-oss-select-0.1.0-SNAPSHOT/jaxb-impl-2.2.3-1.jar
-rw-r--r-- root/root 458739 2018-10-30 16:11 spark-2.2.0-oss-select-0.1.0-SNAPSHOT/jersey-core-1.9.jar
-rw-r--r-- root/root 147952 2018-10-30 16:11 spark-2.2.0-oss-select-0.1.0-SNAPSHOT/jersey-json-1.9.jar
-rw-r--r-- root/root 788137 2018-10-30 16:11 spark-2.2.0-oss-select-0.1.0-SNAPSHOT/aliyun-java-sdk-ecs-4.2.0.jar
-rw-r--r-- root/root 153115 2018-10-30 16:11 spark-2.2.0-oss-select-0.1.0-SNAPSHOT/jdom-1.1.jar
-rw-r--r-- root/root 65437 2018-10-31 14:41 spark-2.2.0-oss-select-0.1.0-SNAPSHOT/aliyun-oss-select-spark_2.11-0.1.0-SNAPSHOT.jar

  • 進(jìn)入${CDH_HOME}/lib/spark/jars目錄，執(zhí)行如下命令：
     
    [root@cdh-master jars]# pwd
    /opt/cloudera/parcels/CDH/lib/spark/jars
    [root@cdh-master jars]# rm -f aliyun-sdk-oss-2.8.3.jar
    [root@cdh-master jars]# ln -s ../../../jars/aliyun-oss-select-spark_2.11-0.1.0-SNAPSHOT.jar aliyun-oss-select-spark_2.11-0.1.0-SNAPSHOT.jar
    [root@cdh-master jars]# ln -s ../../../jars/aliyun-java-sdk-core-3.4.0.jar aliyun-java-sdk-core-3.4.0.jar
    [root@cdh-master jars]# ln -s ../../../jars/aliyun-java-sdk-ecs-4.2.0.jar aliyun-java-sdk-ecs-4.2.0.jar
    [root@cdh-master jars]# ln -s ../../../jars/aliyun-java-sdk-ram-3.0.0.jar aliyun-java-sdk-ram-3.0.0.jar
    [root@cdh-master jars]# ln -s ../../../jars/aliyun-java-sdk-sts-3.0.0.jar aliyun-java-sdk-sts-3.0.0.jar
    [root@cdh-master jars]# ln -s ../../../jars/aliyun-sdk-oss-3.3.0.jar aliyun-sdk-oss-3.3.0.jar
    [root@cdh-master jars]# ln -s ../../../jars/jdom-1.1.jar jdom-1.1.jar
    • 對比測試測試環(huán)境：使用spark on yarn進(jìn)行對比測試，其中Node Manager節(jié)點(diǎn)是4個，每個節(jié)點(diǎn)最多可以運(yùn)行4個container，每個container配備的資源是1核2GB內(nèi)存 。
    測試數(shù)據(jù)：共630MB，包含3列，分別是姓名、公司和年齡 。
     
    ot@cdh-master jars]# hadoop fs -ls oss://select-test-sz/people/
    Found 10 items
    -rw-rw-rw-163079930 2018-10-30 17:03 oss://select-test-sz/people/part-00000
    -rw-rw-rw-163079930 2018-10-30 17:03 oss://select-test-sz/people/part-00001
    -rw-rw-rw-163079930 2018-10-30 17:05 oss://select-test-sz/people/part-00002
    -rw-rw-rw-163079930 2018-10-30 17:05 oss://select-test-sz/people/part-00003
    -rw-rw-rw-163079930 2018-10-30 17:06 oss://select-test-sz/people/part-00004
    -rw-rw-rw-163079930 2018-10-30 17:12 oss://select-test-sz/people/part-00005
    -rw-rw-rw-163079930 2018-10-30 17:14 oss://select-test-sz/people/part-00006
    -rw-rw-rw-163079930 2018-10-30 17:14 oss://select-test-sz/people/part-00007
    -rw-rw-rw-163079930 2018-10-30 17:15 oss://select-test-sz/people/part-00008
    -rw-rw-rw-163079930 2018-10-30 17:16 oss://select-test-sz/people/part-00009進(jìn)入到${CDH_HOME}/lib/spark/，啟動spark-shell ，分別測試使用OSS Select查詢數(shù)據(jù)和不使用OSS Select查詢數(shù)據(jù)：
     
    [root@cdh-master spark]# ./bin/spark-shell
    WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/spark) overrides detected (/opt/cloudera/parcels/CDH/lib/spark).
    WARNING: Running spark-class from user-defined location.
    Setting default log level to "WARN".
    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
    Spark context Web UI available at http://x.x.x.x:4040
    Spark context available as 'sc' (master = yarn, app id = application_1540887123331_0008).
    Spark session available as 'spark'.
    Welcome to
    ______
    / __/_____ _____/ /__
    _ / _ / _ `/ __/'_/
    /___/ .__/_,_/_/ /_/_version 2.2.0-cdh6.0.1
    /_/

    Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_152)
    Type in expressions to have them evaluated.
    Type :help for more information.

    scala> val sqlContext = spark.sqlContext

    推薦閱讀