Kubernetes - Namespace

Default namespace created by Kubernetes 

• kube-system
• kube-public
• default

How it is helpful ?

• These different namespace 
• can be used by same cluster for DEV and PROD environment.
• they will have isolated resources between them so that DEV will have their own resource and PROD will have their own resource.
• they will have their own policies that define who can do what
• Each namespace has a max limit of resources and are not allowed to use more than its allowed limit. 
• We can define policy such that for example, while working in DEV environment, we can't modify a resource of PRODUCTION.

Example usage of custom namespace. 

• Dev
• Test
• Prod

How to name resources of a namespace?

• Same namespace 
• Resources in same namespace ca access via db-service
• Different namespace
• Resources in other namespace can access resources in other namespace via ( say DEV resources via - db-service.dev.svc.cluster.local 

Name details

db-service.dev.svc.cluster.local 

• db-service - Name of the service
• dev - namespace
• svc - service
• cluster.local - domain.

How to create Namspace Or Pod Inside Namespace

• Create in default namespace
• kubectl create -f pod-definition.yml -f
• Create a particular namespace
• kubectl create -f pod-definition --namespace dev
• kubectl create -f pod-definition -n dev
• Create a Namespace
• kubectl create -f namespace-dev.yaml
• kubectl create namespace dev
• Sample Yaml file for Namespace
• apiVersion: v1
• kind: Namespace
• metadata:
• name: dev
• Create POD under a namespace through Yaml file
• apiVersion: v1
• kind: Pod
• metadata
• name: myapp-pod
• namespace: dev
• label
• app: myapp
• type: front-end
• spec
• container:
• name: nginx-container
• image: nbinx
• Accessing Namespace
• Scenario -1 : Access default namespace
• kubectl get pods 
• kubectl get pods --namespace dev
• kubectl get pods --namespace prod

• Change the default namespace to namespace = dev
• kubectl config set-context  $(kubectl config current-context) --namespace dev
• kubectl get pods  <<<< This will show Pods from namespace = DEV.
• kubectl get pods --namespace default << To access default namespace
• kubectl get pods --namespace prod << To access PROD namespace
• Resource Quota
• Resources can be limited in a namespace.
• apiVersion: v1
• kind: ResourceQuota
• metadata:
• name: compute-quota
• namespace: dev
• spec:
• hard:
• pods: "10",
• requests-cpu: "4"
• requests-memory: 5Gi
• limts-cpu: "10"
• limits-memory: 10Gi

Read More

Kubernetes- Deployment

- Defines how to deploy application in production environment.

Scenarios

• Old instance running - Multiple Pod having applications are running in production environment.
• New version of application is available - need to upgrade seamlessly- without any downtime.
• Rolling update
• For this we don't want to upgrade all of them at once as this may impact accessibility. 
• One by one upgrade - And we might want to upgrade one after other.
• In case of failure - we should be able to rollback the upgrade.
• Multiple change to environment - 
• Ex - Scaling the environment, Modifying resource allocation etc.
• And this change should be done for all PODS together and then activated for all Pods as the same time.

Kubernetes Deployment

• Comes at the top of the hierarchy.
• Comes at top of Pod and Replica Set.

• Available with
• Rolling update
• Undo changes
• Pause changes
• Resume changes

How to create deployment

Yaml file 

• same as ReplicaSet except for change in KIND.

apiVersion: apps/v1   

kind: Deployment      << Only change done for Deployment

metadata:

  name: myapp-replicaset

  labels:

    app: myapp

    type: front-end

spec:

  template:   <<< Here need to provide template for POD. We just to insert metadata + spec section of POD.

    metadata:

      name: myapp-pod

      labels:

        app: myapp

        type: front-end

    

    spec:   

      containers: 

        name: nginx-container

        image: nginx

  replicas: 3   <<< This is sibling of template

  selector:        

    matchLabels:

      type: front-end

Commands

Read More

 Kubernetes Architecture, POD, ReplicaSet & Replica Controller

1. Introduction

What is Node ? 

•  Node is a machine - Physical or Virtual on which Kubernetes is installed,
• Node is where containers will be launched by Kubernetes.

What will happen to Nodes in which application is running fails.

• Application goes down -- Means Kubernetes goes down.

Why do we need to have more than one node OR Why do we need CLUSTER?

• Because of one node on which Kuberneres is installed goes down then application will also go down. So for redundancy, we need to have more than one node.
• So we have more than one node grouped together.
• So even if one node goes down, application is still accessible from other nodes.

Now we have CLUSTER ? 

- So who is managing cluster ? 

- Where is the information about members of clusters stored ?

- How are nodes monitored ?

- When a node failed, how do u move workload of one node to other worked node ?

• This is the work of Master node. ( M, S, M, F )
• Manage, Store, Monitor, Failure Handling

When u install Kubernetes, what components are installed?

•  Api server
• Acts as Front end for kubernetes
• User, Management Devices, Command Line interfaces
• talk to API server to interact with Kubernetes cluster.
• etcd
• distributed, reliable key-value storage to store
• all data used to manage cluster.
• Also implements locks in the cluster, so that there is no conflict
• 
  
• kubelet
• agent that runs on each node of the cluster.
• makes sure that conatiners are running on nodes as expected.
• container runtime
• Underlying software that run containers.
• For example - docker.
• controller
• Brain behind orchestration.
• Responds when nodes, containers or end points goes down.
• Take decision to bringup new containers in case of failure.
• schedular
• For distributing work across containers on various nodes.

Master & Worker Nodes

How some node became master and some node became worker nodes ?

- Master node that has 

•     Kube-API server, becomes master node.
• All information is stored in KEY-VALUE ( etcd ) store in master.
• Master also has
• controller and 
• schedular

- Worked node has 

•     kubelet agent, so it becomes worker node.
• Through kubelet-agent worker node can interact with MASTER.
• TO provide health information of worker node.
• And CARRY out instruction from MASTER.

What is kubectl

- This is KUBE command line control

• kubectl run
• to install cluster
• kubectl cluster-info
• View information about cluster
• kubectl get nodes 
• To list all nodes of the cluster

Docker Vs Container

Docker 

- container solution.

Kubernetes 

• kubernetes came to orchestrate docker. and they were tightly coupled.
• But later Kubernetes provided a layer Container Runtime Interface ( CRI ) that can take into any container solution ex - docker, rkt, as long as they adhere to OCI standard (Open Container standard)
• CRTICTL 
• This is command line interface (CLI command) for CRI
• This will work for any container intarface.

POD

Steps - 

- Application is built

- Has been put in docker image

- And Kubernetes can pull the image.

Aim 

- Deploy APplication

- In the form of containers

- On a set of machines

- That are configured as worker nodes.

- In a cluster

But

- Kubernetes does not deploy container directly, it has to be put in POD.

Scale up/down

- We can't add new container inside POD

• We need to create new POD in same node.
• We can create new POD and new Node.

Concept of multi-container pod

• POD can have multiple container.
• But generally they dont have multiple container of same kind
• We can have hlper container
• Ex - Processing file, 

YAML with Kubernetes

Creating POD using YAML based configuration file

Command to create a POD

kubectl create -f pod.yml

Mandatory Values in K8 yaml file

- apiVersion

    - Not a KEY/VALUE, we just pass a string

    - API version that we are using to create Kubernetes

        - Example value - v1 , app/v1

- kind 

    - Not a KEY/VALUE, we just pass a string

    - type of objext we are trying to create 

        - Example - POD, Service, ReplicaSet, Deployment

- Metadata

   - This is dictionary.

   - This is data of the object.

       -- Example - Name, Label etc

       -- Here 'name' and labels are dictionary. (have KEY/Value)

    - NAME and IMAGE are sibling

    - Under "MetaData" we can have only "name" and "label"

        - But under "label" we can define any "key". "value" pair.

- Spec

   -  'Spec' is a dictionary, So Add property under it.

   - Below 'spec', there is 'container'. 'container' is also a list/array 

         - because there can be multiple containers in a pod.

         - So first element in a list indicate that it is first item in the list ( - name )

   - Here name & spec is element inside 'container'

Replica Sets / Replica Controllers.

Replica Controllers

- Are the brain behind Kubernetes

- Have processes that

    - Monitor Kubernetes objects

    - And manage accordingly.

Question  - What is a replica and why do we need replication-controller ?

•     Suppose we have a node + POD + Container running our application
•     What will happen if our application crashes ?
•     So users will no longer to access our application.

Solution

 - If we don't want any disruption, we will need to have multiple POD so that there will not disruption. Even if one fails, other POD will still be accessible.

    Replication Controller 

•   helps running multiple instance of same POD in a Kubernetes cluster.

Two reasons for Replication Controller

• High Availability
•   And so it provides high availability.
•   So if one pod goes down, replication controller quickly instantiates another pod.

• 
  
  

• Load Balancing
• If demand increases, replication controller creates Pods
• On same Node
• Or No multiple Nodes.

Two terms - Replication Controller vs Replica Set

There is minor difference between both but more or less they are same.

• Replication Controller - older way
• Replica Set - newer way.

How to create Replication Controller - using Yaml file.

apiVersion: v1

kind: Replication Controller

metadata:

  name: myapp-rc

  labels:

    app: myapp

    type: fromt-end

spec:   spec defines what in inside the KIND. As POD is inside Replication Controller. So provide template of the Pod.

  template:   <<< Here need to provide template for POD. We just to insert metadata + spec section of POD.

    metadata:

      name: myapp-pod

      labels:

        app: myapp

        type: front-end

    

    spec:   

      containers: 

        name: nginx-container

        image: nginx

But we have something still missing. 

• We have not specified how many PODs to be created? 
• Or how many replicas of POD to be available all the time?
• Solution : Add 'replicas: 3

apiVersion: v1

kind: Replication Controller

metadata:

  name: myapp-rc

  labels:

    app: myapp

    type: fromt-end

spec:   spec defines what in inside the KIND. As POD is inside Replication Controller. So provide template of the Pod.

  template:   <<< Here need to provide template for POD. We just to insert metadata + spec section of POD.

    metadata:

      name: myapp-pod

      labels:

        app: myapp

        type: front-end

    

    spec:   

      containers: 

        name: nginx-container

        image: nginx

replicas: 3   <<< This is sibling of template

How to create ReplicaSet - using Yaml file.

apiVersion: apps/v1    << Note for ReplicationController this is - v1

kind: ReplicaSet

metadata:

  name: myapp-replicaset

  labels:

    app: myapp

    type: front-end

spec:

  template:   <<< Here need to provide template for POD. We just to insert metadata + spec section of POD.

    metadata:

      name: myapp-pod

      labels:

        app: myapp

        type: front-end

    

    spec:   

      containers: 

        name: nginx-container

        image: nginx

replicas: 3   <<< This is sibling of template

selector:           << Only got replicaSet

  matchLabels:

    type: front-end

selector

Defines which POD fall in it.We can define multiple labels.

• Can have labels of PODS define in the same definition
• Can have labels defined somewhere else.

Labels & Selectors

• Monitor POD of specified LABEL specified as part of matchLabels
• Always keep the 3 replicas of the POD instance.
• ReplicaSet is 'process' that monitors the PODS and if any POD fails, it creates new POD.

How ReplicaSet knows which POD to monitor ?

• This is done by labels provided as part of matchLabels.

How to change replicas ?

Two commands

• replace
• scale

Read More

8. Data Analysis - Visualization with Matplotlib

 Basic line plot

%matplotlib notebook -- to create visualization in jupyter notebook

import matplotlib.pyplot as plt

import numpy as np

X = np.arange(10)

plt.plot(X)

Create figure and Plot in two lines

1. Create Figure

fig = plt.figure()

2. Create Plot and add plot to figure

ax1 = fig.add_subplot(2,2,1)  --- 2 rows, 2 column and we are selecting 1st plot.

Without creating subplot ( plt.plot)

fig = plt.figure()

plt.plot(np.random.rand(50), 'k-') -- It will go to rightmost bottom.

Creating figure and axis in the same line.

fig, axes = plt.subplots(2,3)

axes[0,1].hist()

Creating same X and Y axis for all subplots.

fig, axes = plt.subplots(2,3, sharex=True, sharey = True)

Remove space between subplots.

fig, axes = plt.subplots(2,3, sharex=True, sharey = True, wspace =0, hspace=0)

Adding color and linestyle.

fig, axes = plt.subplots(2,3, sharex=True, sharey = True, wspace =0, hspace=0, linestyle ="--" , color ='r')

fig, axes = plt.subplots(2,3, sharex=True, sharey = True, wspace =0, hspace=0, "r--") -- short form.

Add Marker

fig, axes = plt.subplots(2,3, sharex=True, sharey = True, wspace =0, hspace=0, linestyle ="--" , color ='r', marker = "o")

Connecting 2 dots

fig, axes = plt.subplots(2,3, sharex=True, sharey = True, wspace =0, hspace=0, linestyle ="--" , color ='r', marker = "o", drawstyle="steps-post") -- steps-pre/steps-mid/steps

Add label

fig, axes = plt.subplots(2,3, sharex=True, sharey = True, wspace =0, hspace=0, linestyle ="--" , color ='r', marker = "o", drawstyle="steps-post", label="line")

X and Y axis label

fig = plt.figure()

ax1= fig.add_subplot(1,1,1)

ticks = ax1.set_xticks([0,10,20,30])

labels = ax1.set_xticklabels(["zero,", "ten","twnety","thirty"])

ax1.plot()

Change orientation and size of the X labels

fig = plt.figure()

ax1= fig.add_subplot(1,1,1)

ticks = ax1.set_xticks([0,10,20,30])

labels = ax1.set_xticklabels(["zero,", "ten","twnety","thirty"], rotation = 90, fontsize="large")

ax1.plot()

X-axis label, Title

ax1.set_xlabel("Xlabels")

ax1.set_title("Title)

Plots

veritical barplot

ax1.bar(["Car", "Truck", "Bus", "Auto"], [10,20,30,40]) -- Categorical + numeric data

horizontal bar plot

ax1.barh(["Car", "Truck", "Bus", "Auto"], [10,20,30,40]) -- Categorical + numeric data

histogram

ax1.hist(X, bin = 50)

pie chart

ax1.pie([10,20,30], labels=["car", "bus", "truck"])

scatter plot

ax1.scatter(x,y, marker="^", color="g")

Box/Violon plot

ax1.boxplot(X)

ax1.violinplot(X)

Read More

1. Python Machine learning - Regularized Linear Model

Linear Regression

from sklearn.linear_model import LinearRegression     

lin_reg = LinearRegression()

lin_reg.fit(X, y)

print(lin_reg.intercept_, lin_reg.coef_)

lin_reg.predict(X_new)

Polynomial Regression

Transform to polynomial feature

from sklearn.preprocessing import PolynomialFeatures

poly_features = PolynomialFeatures(degree =2, include_bias=False)

X_poly = poly_features.fit_transform(X) --- X_poly will have two terms - degree =1 and degree 2

Now use Linear Regression

lin_reg = LinearRegression()

lin_reg.fit(X_poly, y)

print(lin_reg.intercept_, lin_reg.coef_)

Training and test error

from sklearn.metrics import mean_squared_error

from sklearn.model_selection import train_test_split

def plot_learning_curves(model, x, y):

      X_train, X_val, Y_train, y_val = train_test_split(x, y, test_size=0.2)

      train_error, val_errors = [], []

      for m in range(1, len(X_train)):

             model.fit(X_train[:m], y_train[:m])

             y_train_predict = model.predict(X_train[:m])  -- no of sampling of training is changing from 1 to len(X_train)

             y_val_predict = model.predict(X_val) -- always taken for all validation samples.

              train_errors.append(mean_squared_error(y_train[:m], y_val_predict[:m]))

              val_errors.append(mean_squared_error(y_val, y_val_predict))

plt.plot(np.sqrt(train_errors, "r-+", linewidth=2, label="train"))

plt.plot(np.sqrt(val_errors, "b-+", linewidth=2, label="validation"))           

Using Pipeline

from   sklearn.pipeline import Pipeline

polynomial_regression = Pipeline ([

                                           ("poly features", PolynomialFeatures(degree=10, include_bias=False)),

                                           ("lin reg", LinearRegression())

                                            ])

plot_learning_curve(polynomial_regression, X, y)

Gradient Regression

Batch Gradient Descent

Stochastic Gradient Descent

Mini Batch Gradient Descent

Ridge Regression

from sklearn.linear_model import Ridge

ridge_reg = Ridge(alpha=1, solver="cholesky")

ridge_reg.fit(X,y)

ridge_reg.predict([1.5])

Lasso Regression ( Lest Absolute Shrinkage and Selection Operator Regression )

from sklearn.linear_model import Lasso

lasso_reg = Lasso(alpha = 0.1)

lasso_reg.fit(X, y)

lasso_reg.predict([1.5])

Elastic net

from sklearn.linear_model import ElasticNet

elastic_net = ElasticNet(alpha=0.1, l1_ratio=0.5)

elastic_net.fit(X,y)

elastic_net.predict([1.5])

Read More

7. Data Analysis - Data Aggregation & Grouping

 Split, Apply, Combine

Mean

• X["Values1"].groupby([X["Keys1"]]).mean() -- Mean of "Values1" based on Keys = "Key1"
• X["Values1"].groupby([X["Keys1"],X["Keys2"]]).mean()
• X.groupBy["Keys1"].mean()
• X.groupBy(X["Keys1"], X["Keys2"]).mean()

Count

• X.groupBy(X["Keys1"], X["Keys2"]).size()

GroupBy Clause with FOR Loop

• for name, group in X.groupby([["Key1"]])
• print (name)
• print(group)

Accessing each group in dictionary key

• dict(list(X.groupby([["Key1"]])))

Extracting based on column

• X.groupby(X.dtypes, axis =1) -- column will be seperated based on data types.

Column Grouping

• Create a mapping from column to grouping
• mapColumns={'a':'Group1', 'b':'Group1', 'c':'Group2', 'd':'Group2', 'e':'Group3', 'f':'Group3'}
• X.groupby(mapColumns, axis=1).mean()

Passing Lambda function in groupby

•  X.groupby(lambda x:x<'c', axis =1).mean() -- Two group will be created with TRUE/FALSE column.

Aggregate function

• def max_min(x)
• return x.max() - x.min()
• X.agg([max, min, max_min]) -- For each column min, max, max_min function will be called.
• 
  

Aggregate function with groupby

• X1 = X.groupby(["key1"]) -- Will create dataframe for each groups in "key1"
• X1.agg(min, max, max_min)) -- Will apply aggregate function for each of that column.

Aggregate function with custom defined column name

• X1.agg(("Maximum", max),("Minimum", min),("Maximum_Minimum",max_min))

GroupBy with Apply

• X.groupby().apply(fName)
• X.groupby(["key1"]).apply(lambda x: x.min())  -- Passing lambda function, User defined function can also be applied.

GroupBy with Apply with passing arguments to function

• def minimumFb(X, x):
• return(X.max()-X.min() > x)
• X.groupby(["key1"]).apply(minimumFb(10))

Apply function with bucket analysis

• quartiles = X.cut(X.Values1, 2)
• X.groupby(quartiles).apply(lambda x:x.min())

Pivot Table

• X.pivot_table(values="AvgSizeOfTrip" , index = "Gender" , columns="Group" )

Pivot table with aggregation

• X.pivot_table (values="AvgSizeOfTrip" , index = "Gender" , columns="Group", aggfunc=np.sum)
• X.pivot_table (values="AvgSizeOfTrip" , index = "Gender" , columns="Group", aggfunc={"Col1":np.sum, 'Col2':np.mean}) - different column with different operation.

Pivot table with count

• pd.crosstab(X.gender, X.group)  -- This just counts the frequency.

Read More

6. Data Analysis - Data Wrangling

 Hierarchical Indexing in Pandas Series

• Create multi level indexing in pandas series
• pd.Series(np.random.rand(9), index=[['a','b','c','d','a','b','c','d','a'],[1,2,4,4,5,2,7,8,8]],)
• Access the values
• X['a']  -- using exact index
• X[:, 1]  -- using slice
• X['a':'b', 1] -- using slice
• Access Index
• X.index  -- return tuple
• Changing multi level indexing to row/col 
• X.unstack()   -- to row/col representation.
• X.unstack(level = -1) -- first level index will be ROW and second level index will be column
• X.unstack(level=0) - first level index will be Column and second level index will be row
• Changing dataframe to single pandas series
• X.stack() -- to multi level index, row will become 1st level index and column will become second level index.

Hierarchical indexing in Pandas Dataframe

• Create multi level indexing for both row and columns
• X=pd.Series(np.random.rand(9, 6), index=[['a','b','c','d','a','b','c','d','a'],[1,2,4,4,5,2,7,8,8]],
•                                                 columns=[["col1","col1","col2","col2","col3","col3"],['c1','c2', 'c3', 'c4', 'c5', 'c6']])
• Access name of rows and columns
• X.index.names
• X.columns.name
• Access row/column values
• X.loc[["row1"],[["col1", "col2"]]]

Swap multilevel index

• X1.swap_level(['Row", "Row2"])  -- Row1 and Row2 will interchange.

Sort index

• X1.sort_index(level=0)
• X1.sort_index(level=1)

Aggregation of data

• X1.sum(level = 'Row1')  -- Summation for Row1
• X1.sum(level - 'Row2')
• X1.sum(level = "ColName1", axis =1) -- Summation for Colname1
• X1.sum(level = "ColName2", axis =1)

Changing column to index

• X.set_index(['b', 'a']) 
• X.set_index(['b', 'a'], drop=False)

Change index to column

• X.reset_index()

Merging data sources

If have common column - INNER JOIN

• pd.merge(df1, df2 ) -- should have common column and other column are arranges as seperate column. (OR)
• pd.merge(df1, df2, how="inner")

If does not have common column

• pd.merge(df1, df2 , left_on="lkey") 
• pd.merge(df1, df2 ,  right_on="lkey1") 

OUTER JOIN

• pd.merge(df1, df2, how="outer")  -- INNER JOIN + uncomon element.

LEFT/RIGHT Join

• pd.merge(df1, df2, how="left")  -- INNER JOIN + uncommon element of left dataframe
• pd.merge(df1, df2, how="right")  -- INNER JOIN + uncommon element of right dataframe.

It is possible to have more than one column as key.

• pd.merge(df1, df2 ,  right_on=["lkey1", "lkey2"]) -- 

Concatenate

• pd.concatenate([X, Y], axis=1) -- merged side by side
• pd.concatenate([X,Y], axis =0) -- stacked top to bottom.
• pd.concatenate([X,Y], ignore_index=True) - is stacked top to bottom ignoring matching of row index and pandas creates own index like 0,1,2 etc
• pd.concatenate([X,Y], ignore_index=False) - existing index names of original dataframe is retained.

Combining two dataframe to fill missing values

• X1.combine_first(X2) -- Value of X2 will be used to fill the NAN value in dataframe = X1.

Pivot and Melt function

• X.pivot(index="c1", columns="c2", value = "col3") -- 'c1' becomes ROW, 'c2' becomes column and 'col3' becomes value across 'c1' and 'c2'
• X.pivot(index="c1", columns="c2", value = ["col3","col4"])  -- 'col3' and 'col4' values are stacked side by side in the o/p

Melt function

• pd.melt(X, ["col1"]) -- Three column comes -- "col1", variable(name of all other columns.), values (value of all other columns.)

Read More