Untitled
+TITLE: 大数据
redis
- Redis介绍及常用命令
- Redis资料汇总专题
- books 这些书我都没有下载成功。可能链接已经失效。
- http://www.rediscookbook.org/
- Redis几个认识误区
- Redis系统性介绍 这篇入门级的文章写得不错。
- 从MySQL到Redis,提升数据迁移的效率
- 在线执行redis命令
- Redis的Java客户端Jedis的八种调用方式
- Redis最好的Java客户端jedis入门
- Spring redis
*Kylin
sudo docker pull sequenceiq/kylin:single docker run -P -i -t -h sandbox sequenceiq/kylin:single /etc/bootstrap.sh -bash
Apache spark
安装 Spark SQL
- An Absolutely Unofficial Way to Connect Tableau to SparkSQL (Spark 1.1)
- 在Hadoop-2.2.0集群上安装 Hive-0.13.1 with MySQL
- Apache Spark 1.1.0 on Docker
sbt/sbt -Dhadoop.version=2.2.0 -Phive assembly scp /home/will/下载/apache-hive-0.13.1-bin.tar.gz ipc_zhoumin@192.168.229.72:/home/ipc_zhoumin/app/ (2)下载Hive安装包 stable版:apache-hive-0.13.1-bin.tar.gz tar -zxvf apache-hive-0.13.1-bin.tar.gz (3)配置环境变量 在/etc/profile或者~/.bashrc中配置 #Set Hive Environment export HIVE_HOME=/home/fesh/apache-hive-0.13.1-bin export PATH=$PATH:$HIVE_HOME/bin:$HIVE_HOME/conf 并使配置生效 source /etc/profile sudo su - root ./bin/hive
docker version
离线安装 docker
docker save sequenceiq/kylin >kylin.tar 3.超-超-超级用户 你可能需要一直用超级用户来操作docker,就像早期示例里一直提示的: # 添加docker用户组 $ sudo groupadd docker # 把自己加到docker用户组中 $ sudo gpasswd -a will docker # 重启docker后台服务 $ sudo service docker restart # 注销,然后再登陆 $ exit Wow!连续三个sudo!三次化身“超级用户”,真可谓是“超-超-超级用户”啊!别担心,设置完毕,以后你就再也不用打那么多sudo了! $ wget https://get.docker.com/builds/Linux/x86_64/docker-latest -O docker $ chmod +x docker $ mv docker /usr/bin $ docker -d #启动docker
因为众所周知的原因,在线获取docker的镜像可能有问题,比如说我现在使用cloud focker PaaS,代理也不好使,无法下载镜像。
解决方案:
通过代理下载镜像
https://s3.amazonaws.com/blob.cfblob.com/fee97b71-17d7-4fab-a5b0-69d4112521e6
本地通过以下命令安装镜像
Java代码 收藏代码
cat fee97b71-17d7-4fab-a5b0-69d4112521e6 | docker import - cloudfocker-base
sudo apt-get install docker.io sudo docker pull sequenceiq/spark:1.1.0 docker run -P -i -t -h sandbox sequenceiq/spark:1.1.0 /etc/bootstrap.sh -bash sudo docker run -p 127.0.0.1:10000:10000 -v /home/will:/home/will -i -t -h sandbox sequenceiq/spark:1.1.0 /etc/bootstrap.sh -bash /usr/local/spark/sbin/start-thriftserver.sh & sudo docker ps -l /usr/local/hadoop/bin/hadoop fs -mkdir /tmp /usr/local/spark/bin/beeline !connect jdbc:hive2://localhost:10000 beeline> !connect jdbc:hive2://localhost:10000 scan complete in 22ms Connecting to jdbc:hive2://localhost:10000 Enter username for jdbc:hive2://localhost:10000: root !connect jdbc:hive2://192.168.192.30:10000 Enter password for jdbc:hive2://localhost:10000: 14/10/23 06:28:38 INFO cli.CLIService: SessionHandle [dce9fd44-877f-414b-b2c0-9950f8080392]: openSession() 14/10/23 06:28:38 INFO cli.CLIService: SessionHandle [dce9fd44-877f-414b-b2c0-9950f8080392]: getInfo() Connected to: Hive (version 0.12.0) Driver: Spark Project Core (version 1.1.0) Transaction isolation: TRANSACTION_REPEATABLE_READ CREATE TABLE dimGender ( value string, name string) row format delimited fields terminated by ','; create TABLE factUserInfo ( userId string, name string, gender string) row format delimited fields terminated by ','; LOAD DATA LOCAL INPATH '/home/will/saiku/util/dimgender.data' INTO TABLE dimGender; LOAD DATA LOCAL INPATH '/home/will/saiku/util/factuserinfo.data' INTO TABLE factUserInfo;
saiku data source
Name: Connection Type: URL: jdbc:hive2://localhost:10000 Schema: Jdbc Driver: org.apache.hive.jdbc.HiveDriver Username: root Password:
oozie
kafka
Kylin
docker pull sequenceiq/kylin
$ wget https://github.com/sequenceiq/docker-kylin/blob/master/ambari-functions $ source ambari-functions $ kylin-deploy-cluster 3
#Ambari Dashboard http://<container_ip>:8080 Use admin/admin to login. Make sure HBase is running.
#Kylin Dashboard http://<container_ip>:9080 The default credentials to login are: admin:KADMIN.
docker rm -f $(docker ps -a -q)
hadoop
http://tecadmin.net/setup-hadoop-2-4-single-node-cluster-on-linux/ http://abloz.com/hbase/book.html http://tecadmin.net/install-apache-hive-on-centos-rhel/ https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-InstallingHivefromaStableRelease source ambari-functions&&docker rm -f $(docker ps -a -q) && kylin-deploy-cluster 1 emacs-snapshot &
source ambari-functions&&docker rm -f $(docker ps -a -q)
http_proxy=http://127.0.0.1:8087 kylin-deploy-cluster 1
Presto
./presto --server localhost:8080 --catalog mysql --schema sone