Spark 2 on Hortonworks Data Platform 2.5 Sandbox

Since apparently there are still issues with using Spark2 from Zeppelin, you may want to switch to shell to use Spark2 with HDP 2.5.

Summarizing the flow (learned it the hard way):

Connect via SSH to the sandbox (e.g. Putty on Windows) on port 2122. This will allow you to use copy-paste (vs console in VirtualBox)

username: root

password (default): hadoop

SSH into the docker container that runs HDP

ssh root@172.17.0.2

username: root

password (default): hadoop

Make sure you use Spark2

export SPARK_MAJOR_VERSION=2

Run Spark shell

cd /usr/hdp/current/spark2-client/bin
./spark-shell

Comments

Eclipse + EGit - "The authenticity of host ... can't be established" challenge

Recently while writing new Android code I decided that it's the highest time to have a Git repository not only on my hard drive, but also safe in the Internet. After quick search and finding out that I have accounts at almost every popular service that provides Git hosting, I figured out that one that covers everything I need (wiki, bug tracking, code hosting, forums) is the good old sourceforge. I used it also with no problems few months ago on another mobile project, so I was hoping that pushing code there will be a piece of cake. But then when I tried to do it (after configuring the project on the sourceforge site), I got very interesting error: ssh://USER@git.code.sf.net:22: org.eclipse.jgit.transport.CredentialItem$YesNoType:The authenticity of host 'git.code.sf.net' can't be established. RSA key fingerprint is 86:7b:1b:12:85:35:8a:b7:98:b6:d2:97:5e:96:58:1d. Are you sure you want to continue connecting? In theory it's nothing bad, you press the "Y...

HTTPS for dummies - so how HTTPS really works in 5 mins

What is HTTPS? HTTPS (HyperText Transfer Protocol Secure) is a way of transferring data over internet in a secure manner. It's achieved through adding SSL (Secure Socket Layer)/TLS (Transport Layer Security) on top of standards HTTP. What HTTPS gives us? End-2-end encryption of data - from the browser to the server and back = even if someone reads the data you are sending, they will not be able to understand anything out of it Confirmation of the identity of the website we are accessing = you are sure that the website that looks like your bank is actually your bank (and not a phishing website) How does it work? First you need a pair of SSL certificates: One installed in your web browser (in most cases shipped together with your browser, provided by one of so-called trusted Certificate Authorities) One installed on the website (which is acquired by the website owner) Each of those SSL certificates includes the following information: Public information: name of...

Rendering AdMob view on Canvas (SurfaceView) in android

If you're wondering how put a working AdMob view into your SurfaceView and are tired of looking in the Internet for solution (somehow most suggestions that I found on forums didnt work), here it is... Assumptions: A. We request the ad on creation (you may want to refresh it later though...) B. The AdView is put on the bottom of the screen C. It's a production ready code, but if you want to test it - add testDevices to the adRequest D. You've already set AndroidManifest properly as described in the Getting Started tutorial 1. In the activity that initializes your SurfaceView add a field representing your adView, for example: private AdView adView; 2. In the onCreate method of the same activity put the following code: // window manager preparation WindowManager.LayoutParams windowParams = new WindowManager.LayoutParams(); windowParams.gravity = Gravity.BOTTOM; windowParams.x = 0; windowParams.y = 0; windowParams.height = WindowManager.Lay...

Lech Migdal's blog

Search This Blog