Skip to main content

Spark 2 on Hortonworks Data Platform 2.5 Sandbox

Since apparently there are still issues with using Spark2 from Zeppelin, you may want to switch to shell to use Spark2 with HDP 2.5.

Summarizing the flow (learned it the hard way):
  1. Connect via SSH to the sandbox (e.g. Putty on Windows) on port 2122. This will allow you to use copy-paste (vs console in VirtualBox)

    2. username: root
    password (default): hadoop

  2. SSH into the docker container that runs HDP
    3. ssh root@172.17.0.2

     username: root
    password (default): hadoop

  3. Make sure you use Spark2

    4. export SPARK_MAJOR_VERSION=2

  4. Run Spark shell

    5. cd /usr/hdp/current/spark2-client/bin
    ./spark-shell

Comments

Post a Comment

Popular posts from this blog

Eclipse + EGit - "The authenticity of host ... can't be established" challenge

Recently while writing new Android code I decided that it's the highest time to have a Git repository not only on my hard drive, but also safe in the Internet. After quick search and finding out that I have accounts at almost every popular service that provides Git hosting, I figured out that one that covers everything I need (wiki, bug tracking, code hosting, forums) is the good old sourceforge. I used it also with no problems few months ago on another mobile project, so I was hoping that pushing code there will be a piece of cake. But then when I tried to do it (after configuring the project on the sourceforge site), I got very interesting error: ssh://USER@git.code.sf.net:22: org.eclipse.jgit.transport.CredentialItem$YesNoType:The authenticity of host 'git.code.sf.net' can't be established. RSA key fingerprint is 86:7b:1b:12:85:35:8a:b7:98:b6:d2:97:5e:96:58:1d. Are you sure you want to continue connecting? In theory it's nothing bad, you press the "Y
16 comments
Read more

HTTPS for dummies - so how HTTPS really works in 5 mins

What is HTTPS? HTTPS (HyperText Transfer Protocol Secure) is a way of transferring data over internet in a secure manner. It's achieved through adding SSL (Secure Socket Layer)/TLS (Transport Layer Security) on top of standards HTTP. What HTTPS gives us? End-2-end encryption of data - from the browser to the server and back = even if someone reads the data you are sending, they will not be able to understand anything out of it Confirmation of the identity of the website we are accessing = you are sure that the website that looks like your bank is actually your bank (and not a phishing website) How does it work? First you need a pair of SSL certificates: One installed in your web browser (in most cases shipped together with your browser, provided by one of so-called trusted Certificate Authorities) One installed on the website (which is acquired by the website owner) Each of those SSL certificates includes the following information:  Public information: name of
Post a Comment
Read more

How to make Logitech Trackball Marble Wheel work

If you bought Trackball Marble from Logitech, the first challenge you encounter is probably related to the lack of the wheel button. Unfortunately the software provided with the device for Windows doesn't help (neither Universal or Auto Search aren't really working as I was expecting). Internet suggests mostly one option, app called Marble Mouse Scroll Wheel http://marble-mouse-scroll-wheel.software.informer.com/ To some extent it works, but I wasn't able to make it work in google maps or in picture viewer. Moreover setting where crashing very often (I am running windows 7 64 bits). Fortunately there is a way to have a semi-wheel button behavior with this trackball, but with a different software - X-Mouse Button Control: http://www.highrez.co.uk/downloads/XMouseButtonControl.htm Setup Mouse button 4 and 5 as wheel up and wheel down. Then also update Logitech SetPoint settings to replace the behavior of those button to default. Voila - now you can emulate wheel
2 comments
Read more