Untitled

+TITLE: 大数据

redis

  1. Redis介绍及常用命令
  2. Redis资料汇总专题
  3. books 这些书我都没有下载成功。可能链接已经失效。
  4. http://www.rediscookbook.org/
  5. Redis几个认识误区
  6. Redis系统性介绍 这篇入门级的文章写得不错。
  7. 从MySQL到Redis,提升数据迁移的效率
  8. 在线执行redis命令
  9. Redis的Java客户端Jedis的八种调用方式
  10. Redis最好的Java客户端jedis入门
  11. Spring redis

*Kylin

sudo docker pull sequenceiq/kylin:single

docker run -P -i -t -h sandbox sequenceiq/kylin:single /etc/bootstrap.sh -bash

Apache spark

安装 Spark SQL

  1. An Absolutely Unofficial Way to Connect Tableau to SparkSQL (Spark 1.1)
  2. 在Hadoop-2.2.0集群上安装 Hive-0.13.1 with MySQL
  3. Apache Spark 1.1.0 on Docker
sbt/sbt -Dhadoop.version=2.2.0 -Phive  assembly
scp /home/will/下载/apache-hive-0.13.1-bin.tar.gz ipc_zhoumin@192.168.229.72:/home/ipc_zhoumin/app/

(2)下载Hive安装包

stable版:apache-hive-0.13.1-bin.tar.gz

tar -zxvf apache-hive-0.13.1-bin.tar.gz
(3)配置环境变量

在/etc/profile或者~/.bashrc中配置

#Set Hive Environment
export HIVE_HOME=/home/fesh/apache-hive-0.13.1-bin
export PATH=$PATH:$HIVE_HOME/bin:$HIVE_HOME/conf
并使配置生效

source /etc/profile

sudo su - root 

./bin/hive

docker version

离线安装 docker

docker save sequenceiq/kylin >kylin.tar
3.超-超-超级用户

你可能需要一直用超级用户来操作docker,就像早期示例里一直提示的:
# 添加docker用户组
$ sudo groupadd docker
# 把自己加到docker用户组中
$ sudo gpasswd -a will docker
# 重启docker后台服务
$ sudo service docker restart
# 注销,然后再登陆
$ exit

Wow!连续三个sudo!三次化身“超级用户”,真可谓是“超-超-超级用户”啊!别担心,设置完毕,以后你就再也不用打那么多sudo了!


$ wget https://get.docker.com/builds/Linux/x86_64/docker-latest -O docker
$ chmod +x docker
$ mv docker /usr/bin
$ docker -d #启动docker

因为众所周知的原因,在线获取docker的镜像可能有问题,比如说我现在使用cloud focker PaaS,代理也不好使,无法下载镜像。

解决方案:

通过代理下载镜像

https://s3.amazonaws.com/blob.cfblob.com/fee97b71-17d7-4fab-a5b0-69d4112521e6

本地通过以下命令安装镜像

Java代码 收藏代码

cat fee97b71-17d7-4fab-a5b0-69d4112521e6 | docker import - cloudfocker-base

sudo apt-get install docker.io
sudo docker pull sequenceiq/spark:1.1.0


docker run -P -i -t -h sandbox sequenceiq/spark:1.1.0 /etc/bootstrap.sh -bash

sudo docker run -p 127.0.0.1:10000:10000 -v /home/will:/home/will   -i -t -h sandbox sequenceiq/spark:1.1.0 /etc/bootstrap.sh -bash

/usr/local/spark/sbin/start-thriftserver.sh &


sudo docker ps -l

/usr/local/hadoop/bin/hadoop fs -mkdir /tmp


/usr/local/spark/bin/beeline


!connect jdbc:hive2://localhost:10000
beeline> !connect jdbc:hive2://localhost:10000
scan complete in 22ms
Connecting to jdbc:hive2://localhost:10000
Enter username for jdbc:hive2://localhost:10000: root
!connect jdbc:hive2://192.168.192.30:10000
Enter password for jdbc:hive2://localhost:10000: 
14/10/23 06:28:38 INFO cli.CLIService: SessionHandle [dce9fd44-877f-414b-b2c0-9950f8080392]: openSession()
14/10/23 06:28:38 INFO cli.CLIService: SessionHandle [dce9fd44-877f-414b-b2c0-9950f8080392]: getInfo()
Connected to: Hive (version 0.12.0)
Driver: Spark Project Core (version 1.1.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ



CREATE TABLE dimGender (  value string,  name string) row format delimited fields terminated by ',';

create TABLE factUserInfo (  userId string,  name string,  gender string) row format delimited fields terminated by ',';

LOAD DATA LOCAL INPATH '/home/will/saiku/util/dimgender.data' INTO TABLE dimGender;     

LOAD DATA LOCAL INPATH '/home/will/saiku/util/factuserinfo.data' INTO TABLE factUserInfo;

saiku data source

Name:
Connection Type:
URL: jdbc:hive2://localhost:10000
Schema:
Jdbc Driver: org.apache.hive.jdbc.HiveDriver
Username: root
Password:

kafka

sudo docker pull wurstmeister/kafka

docker run -P -i -t -h sandbox wurstmeister/kafak

Kylin

docker pull sequenceiq/kylin

$ wget https://github.com/sequenceiq/docker-kylin/blob/master/ambari-functions $ source ambari-functions $ kylin-deploy-cluster 3

#Ambari Dashboard http://<container_ip>:8080 Use admin/admin to login. Make sure HBase is running.

#Kylin Dashboard http://<container_ip>:9080 The default credentials to login are: admin:KADMIN.

docker rm -f $(docker ps -a -q)

hadoop

http://tecadmin.net/setup-hadoop-2-4-single-node-cluster-on-linux/ http://abloz.com/hbase/book.html http://tecadmin.net/install-apache-hive-on-centos-rhel/ https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-InstallingHivefromaStableRelease source ambari-functions&&docker rm -f $(docker ps -a -q) && kylin-deploy-cluster 1 emacs-snapshot &

source ambari-functions&&docker rm -f $(docker ps -a -q)

http_proxy=http://127.0.0.1:8087 kylin-deploy-cluster 1

Presto

./presto --server localhost:8080 --catalog mysql --schema sone

Comments

comments powered by Disqus