spark_intallation

  1. 下载

下载spark

1
wget --no-check-certificate https://www.apache.org/dyn/closer.lua/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.7.tgz

下载 JAVA JDK

1
wget --no-check-certificate -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u171-b11/512cd62ec5174c3487ac17c61aaa89e8/jdk-8u171-linux-x64.tar.gz

如果解压出错，可能是问下下载的不完整，试试删了重新下载

安装JDK

将下载的jdk解压到/usr/java

1
2
3
cd /usr
mkdir java
tar -zxvf jdk-8u171-linux-x64.tar.gz -C /usr/java

修改环境变量

在/etc/profile文件末尾加上

1
2
3
4
5
6
#Set java environment
JAVA_HOME=/usr/java/jdk1.8.0_171
JRE_HOME=/usr/java/jdk1.8.0_171/jre
CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export JAVA_HOME JRE_HOME CLASS_PATH PATH

使配置文件生效

1
source /etc/profile

现在java命令能正确相应

1
2
3
4
[root@localhost ~]# java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

scrapy

使用scrapy提交一个表单

一些登录界面提交的参数除了用户名和密码还有一些参数是隐藏的，如
<input type="hidden" value = "aaa">，通过FormRequest.from_response可以自动提取要提交的参数，再添加上用户名和密码

如官网文档中的例子

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import scrapy

class LoginSpider(scrapy.Spider):
    name = 'example.com'
    start_urls = ['http://www.example.com/users/login.php']

    def parse(self, response):
        return scrapy.FormRequest.from_response(
            response,
            formdata={'username': 'john', 'password': 'secret'},
            callback=self.after_login
        )

    def after_login(self, response):
        # check login succeed before going on
        if "authentication failed" in response.body:
            self.logger.error("Login failed")
            return

        # continue scraping with authenticated session...

Octave

1. format display

1
2
3
4
5
>> a = pi;
>> a
a =  3.1416
>> disp(sprintf('2 decimals : %0.2f', a))
2 decimals : 3.14

format long|short ones been set, will work for all next command untill terminal been closed or format again

1
2
3
4
5
6
7
8
9
10
11
12
>> a = pi
a =  3.1416
>> format long
>> a
a =  3.141592653589793
>> b = pi
b =  3.141592653589793
>> format short
>> a
a =  3.1416
>> b
b =  3.1416

2. generator

1
2
3
4
>> v = 1:1:5
v =

   1   2   3   4   5

3. declare a matrix

1
2
3
4
5
6
7
8
9
10
11
12
13
>> matrix = [1 2; 3 4; 5 6]
matrix =

   1   2
   3   4
   5   6

>> matrix = [1,2; 3,4; 5,6]
matrix =

   1   2
   3   4
   5   6

vector is a Nx1 or 1xN matrix

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
>> A = ones(3,3)
A =

   1   1   1
   1   1   1
   1   1   1

>> A = rand(3,3)
A =

   0.87477   0.94144   0.57762
   0.34669   0.54274   0.89587
   0.16017   0.26238   0.86232

>> A = randn(3, 3)
A =

  -0.7941526  -0.0018799   2.0211357
  -0.7958727  -0.3035248   0.9248759
   0.7920713  -2.4600881   0.1848283

>> A = magic(3)
A =

   8   1   6
   3   5   7
   4   9   2

>> C = eye(3)
C =

Diagonal Matrix

   1   0   0
   0   1   0
   0   0   1

4. property and functions of matrix

1. reshape

1
2
3
4
5
6
7
8
9
10
11
12
>> c = [1, 2, 3; 4, 5, 6]
c =

   1   2   3
   4   5   6

>> reshape(c, 3, 2)
ans =

   1   5
   4   3
   2   6

2. size

return a 1x2 matrix

size(c, 1) get size of row
size(c, 2) get size of column

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
>> c = ones(2, 3)
c =

   1   1   1
   1   1   1

>> size(c)
ans =

   2   3

>> size(c, 1)
ans =  2
>> size(c, 2)
ans =  3

3. length

return max of dimension

1
2
3
4
5
6
7
8
9
>> v = [1 2 3 4]
v =

   1   2   3   4

>> length(v)
ans =  4
>> length(c)
ans =  3

4. transpose

1
2
3
4
5
6
7
8
9
10
11
12
13
>> c = magic(3)
c =

   8   1   6
   3   5   7
   4   9   2

>> c'
ans =

   8   3   4
   1   5   9
   6   7   2

5.

.
make function behave on every element

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
>> A = magic(3)
A =

   8   1   6
   3   5   7
   4   9   2

>> A = 1 ./ A
A =

   0.12500   1.00000   0.16667
   0.33333   0.20000   0.14286
   0.25000   0.11111   0.50000

>> A = 2 .* A
A =

   0.25000   2.00000   0.33333
   0.66667   0.40000   0.28571
   0.50000   0.22222   1.00000

6. max

max(A,[],1)

return the max number of every column

max(A,[],2)

return the max number of every row

1
2
3
4
5
6
7
8
9
10
11
>> max(A,[],1)
ans =

   8   9   7

>> max(A,[],2)
ans =

   8
   7
   9

7.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
>> A = magic(3)
A =

   8   1   6
   3   5   7
   4   9   2

>> log(A)
ans =

   2.07944   0.00000   1.79176
   1.09861   1.60944   1.94591
   1.38629   2.19722   0.69315

>> exp(A)
ans =

   2980.9580      2.7183    403.4288
     20.0855    148.4132   1096.6332
     54.5982   8103.0839      7.3891

5. function between matrixs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
>> a = magic(3)
a =

   8   1   6
   3   5   7
   4   9   2

>> a = 2 * a
a =

   16    2   12
    6   10   14
    8   18    4

>> a = [1, 2, 3; 4, 5, 6; 7, 8, 9]
a =

   1   2   3
   4   5   6
   7   8   9

>> b = [7, 8, 9; 4, 5, 6; 1, 2, 3]
b =

   7   8   9
   4   5   6
   1   2   3

>> a .* b
ans =

    7   16   27
   16   25   36
    7   16   27

>> a * b
ans =

    18    24    30
    54    69    84
    90   114   138

6. Miscellaneous

1. convert a matrix to vector

suppose A is a 3x3 Matrix

1
2
3
A = A(:)

reshape(A, 9 ,1)

2. invert a matrix

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
>> A = magic(3)
A =

   8   1   6
   3   5   7
   4   9   2

>> temp = pinv(A)
temp =

   0.147222  -0.144444   0.063889
  -0.061111   0.022222   0.105556
  -0.019444   0.188889  -0.102778

>> temp * A
ans =

   1.00000   0.00000  -0.00000
  -0.00000   1.00000   0.00000
   0.00000   0.00000   1.00000

3. flipud

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
>> A = eye(3)
A =

Diagonal Matrix

   1   0   0
   0   1   0
   0   0   1

>> A = flipud(A)
A =

Permutation Matrix

   0   0   1
   0   1   0
   1   0   0

4.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
>> A = magic(3)
A =

   8   1   6
   3   5   7
   4   9   2

>> B = ones(3)
B =

   1   1   1
   1   1   1
   1   1   1

>> [A; B]
ans =

   8   1   6
   3   5   7
   4   9   2
   1   1   1
   1   1   1
   1   1   1

>> [A B]
ans =

   8   1   6   1   1   1
   3   5   7   1   1   1
   4   9   2   1   1   1