SparkPipeExample/README at master · ImageGuidedTherapyLab/SparkPipeExample · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
Adapted from: http://blog.madhukaraphatak.com/pipe-in-spark/

Usage
=====
$ bsub < spark.lsf


Usage
=====

$ python pipeline.py --pipe
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
16/09/07 16:04:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Running shell script
Running shell script
Running shell script
Running shell script
Running shell script
hi
Running shell script
Running shell script
Running shell script
Running shell script
Running shell script
hello
Running shell script
Running shell script
Running shell script
Running shell script
Running shell script
how
Running shell script
Running shell script
Running shell script
Running shell script
Running shell script
are
Running shell script
Running shell script
Running shell script
Running shell script
you