Follow Us :

  With machine learning, to perform any task, we need to design the right set of features and feed those features to the machine learning model. Feature engineering is a vital task for the success of any machine learning model. But it is hard to engineer the right set of features when dealing with unstructured data like text and images. In those cases, we can use deep learning.

  With deep learning, we are not required to engineer the features since the deep neural network consists of several numbers of hidden layers. It implicitly learns and extracts the right set of features by itself. So, we don’t have to perform feature engineering by ourselves. Thus deep learning is widely used in the task where it is hard to perform feature engineering such as image recognition, text classification, and so on. Thus, in this way, deep learning differs from machine learning.
The artificial neural network consists of one input, N number of hidden, and one output layer. When the artificial neural network consists of a large number of hidden layers then it is often called the deep neural network.
The transfer function is most commonly known as the activation function. It is mainly used for establishing non-linearity in the neural network. That is, it is mainly used for introducing the non-linear transformation in the neural network for learning the intricate patterns in the data.
The sigmoid activation function scales the value between 0 to 1 and it is centered at 0.5 whereas the tanh activation function scaled the value between -1 to 1 and it is centered at 0.
Suppose, x is given as an input to the ReLU function. If the value of x is less than 0 then the ReLU function returns 0 as output. If the value of x is greater than or equal to 0 then the ReLU function returns x as output. Thus, the ReLU function always returns 0 when the value of x is less than 0 (that is when x is a negative value) and this is often referred to as the dying ReLU problem.
To combat the dying ReLU problem, we use the Leaky ReLU. With the leaky ReLU activation function, we introduce a small slope for a negative value. That is, instead of always returning the 0 every time when x is a negative value, leaky ReLU returns the x multiplied by a small number called alpha. We usually set the value of alpha to 0.01. Thus, leaky ReLU returns x when x is greater than or equal to 0 and it returns x multiplied by alpha when x is less than 0.
In the leaky ReLU, instead of setting the alpha to a small value, we can feed the alpha as a parameter to the network and let the network learn the optimal value of the alpha, and this type of leaky ReLU is often called the parametric ReLU function. In the leaky ReLU, we can also set the random value to the alpha. When the value of alpha is set to some random value it is often called the randomized ReLU function.
The softmax function converts the given inputs to the range of 0 to 1. It acts as a generalization of the sigmoid activation function. When we perform a classification task, it is more desirable to apply the softmax function in the output layer since it returns the probability of each class is the output.
Batch size is often set to the power of 2 such as 32, 64, 128, and so on based on the memory requirements of CPU or GPU.
One iteration implies that our network has seen a batch of data points once whereas one epoch implies that our network has seen all the data points once.
The number of neurons in the input layer is set to the number of inputs. The number of neurons in the output layer is set based on the task we perform. For instance, if we are performing regression, then we can set the number of neurons in the output layer to 1. If we are performing classification, then we can set the number of neurons in the output layer to several classes present in the dataset where each neuron emits the probability of the corresponding class being an output.
The number of neurons in the hidden layer can be set based on any of the following methods:

- The number of hidden neurons should be between the size of the input layer and the size of the output layer
- The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer
- The number of hidden neurons should be less than twice the size of the input layer
However, there is still not yet a standard right rule to set the number of neurons in the hidden layer.
Data augmentation is widely used for increasing training data. Suppose, we are training the network to perform an image classification task and we have only less number of images in our training set and we have no access to obtain more images to include in the training set. In that case, we can perform data augmentation by cropping, flipping, and padding the images and obtain the new images and include them in our training set.
It is not a good practice to initialize all the weights with zero. During backpropagation, we train the network by calculating the gradients of the loss function with respect to the weights. When we set all the weights to zero, then the derivatives will have the same value for all the weights. This makes the neurons to learn the same feature. Thus, when we set all the weights to zero, then we end up with a network where all the neurons learn the same feature.
Some of the most common reasons for the loss not to decrease during training the network includes when we stuck at a local minimum when we set the learning rate to a low value when the regularization parameter is high.
Some of the hyperparameters of the network include the following:
  • Number of neurons in the hidden layer
  • Number of hidden layers
  • The activation function in each layer
  • Weight initialization
  • Learning rate
  • Number of epochs
  • Batch size
We train the network by performing backpropagation. During backpropagation, we apply any optimization method and find the optimal weights. Gradient descent is the most commonly used optimization method while training the network during backpropagation.
Some of the methods used for preventing overfitting in neural networks include the following:
  • Dropouts
  • Early stopping
  • Regularization
  • Data augmentation
Some of the most common reasons for the loss leading to nan during training the network includes when the learning rate is set to a high value when the gradient blows up and improper or poor loss function.
Data normalization is usually performed as a preprocessing step. Data normalization implies we normalize the data points by subtracting the mean of each data point and dividing by its standard deviation. It helps in attaining better convergence during training the network.
Some of the most commonly used weight initialization methods include random initialization and Xavier initialization.
While training the deep network, the distribution of the hidden units activation value changes due to the change in weights and bias. This leads to the problem called the internal covariate shift and causes the training time to slow down. We can avoid this problem of internal covariate shift by applying batch normalization. Batch normalization as the name suggests denotes the normalizing hidden units activation value. It also helps in reducing the training time of the network.
Early stopping is often used to control overfitting. With early stopping, we stop the process of training the neural network before the weights have converged. That is, we check the performance of our network on the validation set which is not used for the training. When the performance of the network has not been improved over the validation set then we stop training the network.
Dropout often referred to the dropping off some of the neurons in the neural network. That is, while training the network we can ignore certain neurons randomly and this helps is preventing the network from over fitting to the training data.
There is not any standard and optimal way to decide the number of hidden layers. We can choose the number of hidden layers based on the intuitiveness obtained from the problem we are dealing with.
For a simple problem, we can build the network with 2 or less than 2 hidden layers and for a complex problem, we can build a deep network with many hidden layers. As specified earlier, there is no rule of thumb in deciding the number of hidden layers.
Following are the steps involved:
a. Get data from various data sources available.
b. Generate research question from data.
c. Identify variables present in data. Also, identify important variables or variables to be analyzed as such.
d. Generate hypothesis.
e. Analyze data using graph data like histogram for example.
f. Fit a model from analyzed data.
g. Accept or reject the hypothesis.
h. Research question answer found.
Anything which keeps on changing is called variable. Variables are of different type and below are the following:
Dependant/Outcome: A variable being affected, for example annual temperature in above example.
Independent/Predictor: A variable affecting the outcome for e.g. deforestation, pollution, and so on in above example
Binary variables are those in which only two classes exist, like live or dead male or female on or off.
Univariate variable: When the variable under consideration is only one then it is called univariate variable study.
Bivariate variable: Involves study of relationship between two variables.
The discrepancy between the measured value and actual value in terms of number is called measurement error.
For Example While buying fruits from a vendor in kilograms, if we wanted 1 kilogram of fruits and the vendor’s weighing machine showed 1 kilogram when we brought the same. After checking the same in another machine, if the measured value shows 0.1 kilogram less than expected then this difference is what we call as measurement error.
Validity implies whether an instrument measures what it is supposed to measure.
Reliability implies whether the instrument gives consistent result across different conditions.
For example, if we test the same value twice on the same entity then the results from the instrument should remain same if it has to be reliable. Such tests are known as test-retest.
There are two ways in which hypotheses can be tested:
a. Correlational research
This is also known as cross-sectional research This involves observing the natural pattern or occurrence to test Original occurrences are not manipulated
b. Experimental research
We select the variables of interest Then we manipulate some aspect of the environment Observe the effect on the selected variable
There are two types in variation explained as follows:
Systematic variation:
Introduced by experimenter
The participants are tested under different conditions and the difference in condition is introduced by experimenter
For Example to test use of woolen clothes w.r.t. temperature, we can test a group of 20 people, in both hot and cold climate. Thus, the difference introduced here is in terms of temperature only.
Unsystematic variation:
Introduced by random factors that exist between the experimental conditions.
For Example To test use of woolen clothes w.r.t. temperature, we can test a group of 20 people. Of the selected set some might behave differently than expected due to factors like illness and so on.
Same measure is measured under different conditions on same set participants.
The difference in two conditions can be caused by the following:
The manipulation/changes that was carried out on the participants
Factors that might affect the way in which a participant performs from one time to the next
Nominal measurements are there more than two classes. Such categories can be numbers too.
These are nominal variables which have logical order. Examples include team ranks in cricket or football, merit list of students appearing for grade students.
These are variables which can take can any value on the measurement scale example includes pitch of voice which can take any possible value within the range.
These are variables which can take fixed values in range. For example, number of customers in a bank.
Yes, based upon the motive of study, it is possible to convert discrete values to continuous and vice versa, for example, Level of water in the tank can take any value in the range and as such a continuous variable.
But we can approximate the same to three different levels like empty, full, or half empty and this now becomes discrete in nature.
These are variables which are grouped on interval. Example is age can be divided in range like 10-20, 20-30 and so on and, person with particular age would be placed in one of the above groups. When intervals are equal, they represent difference in equal property being measured.
This is a subtype of interval variables where ratio of scales is used for measurement.
For Example, Water representation in chemistry is H2O which represents two molecules of hydrogen and one molecule of oxygen. Thus, the ratio of elements is 2: 1.
By using randomization we can ensure that any variation introduced, is due to changes in the conditions/variables introduced rather than any other unexpected changes during the process. Thus, it helps in removing other sources of systematic variation.
Same measure is measured under different conditions on different set of participants. The differences between the two conditions can be caused by the following: The manipulation/changes that were carried out on the participants Difference in nature or characteristics of participants in each case
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT * FROM emp WHERE Sal*12 BETWEEN 600000 AND 1000000;
SELECT *
FROM emp e
INNER JOIN dept d ON (e.DeptNo = d.DeptNo)
WHERE d.DName = ‘Account’
DeotNo Dname Loc
50 Service Delhi
51 Account Mumbai
SELECT * FROM dept;
DeptNo Dname Loc
50 Service Delhi
51 Account Mumbai
SELECT DISTINCT(job) FROM dept;
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT COUNT(*) FROM emp;
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT * from emp ORDER BY Sal;
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT * from emp WHERE EmpNo in (SELECT MGR FROM emp);
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT * FROM emp WHERE HireDate < ’01-Jan-2018′;
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT *, sal*12 as Annual_Income, FROM emp ORDER BY Annual_Income;
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT *, months_between(sysdate,HireDate) as Exp FROM emp WHERE EmpNo IN (SELECT MGR FROM emp);
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT EName FROM emp WHERE Comm < Sal;
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT EName FROM emp WHERE (sal/30)>200;
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT * FROM emp WHERE Job = ‘Waiter’;
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT * FROM emp WHERE HireDate IN (’01-Aug-2018′,’04-Aug-2018′,’29-OCt-2018′) ORDER BY HireDate DESC;
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT * FROM emp WHERE HireDate BETWEEN (’01-Jan-2018′) AND (’31-Dec-2018′);
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT * FROM emp WHERE EName LIKE ‘N____’; Or SELECT * FROM emp WHERE EName LIKE ‘N%’ AND len(EName) = 5;
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT * FROM emp WHERE Upper(EName) LIKE ‘__K%’;
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT * FROM emp WHERE to_char(HireDate,’mon’)=’Aug’
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT * FROM emp WHERE to_char(HireDate, ‘yy’) LIKE ‘9_’;
EmpNo EName Job MGR HireDa Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct -18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT * FROM emp WHERE to_char(HireDate,’MON’) NOT IN ‘(‘Oct’);
Categorical measurement contains categories i.e. distinct entities. Example of categories of life on earth is plants, animals, and so on.
Due to the exponentially moving the average of gradients, Adam fails to reach convergence and may reach the sub-optimal solution instead of the globally optimal solution. This happens because when we use the exponentially moving average of gradients, we miss the information about the gradients that occur less frequently. So, to combat this issue, we use AMSGrad.
In Adam, we compute the running average of squared gradients as we compute in RMSProp. But instead of computing only the running average of squared gradients, we also compute the running average of gradients, That is, Adam uses both first and second-order moments of the gradients.
Yes, we can get rid of the learning date using adadelta.
We can set the learning rate adaptively using adagrad. Using adagrad method, we assign a high learning rate when the previous gradient value is low and we assign a low learning rate when the previous gradient value is high. This makes the learning rate to change adaptively based on the past gradient updates.
One issue we encounter with the momentum-based gradient descent method is that it causes us to miss out on the minimum value.
Suppose, we are near to attaining convergence and when the value of momentum is high, then the momentum pushes the gradient step high and we miss out on the minimum value, that is we overshoot the minimum value.
Nesterov accelerated momentum is used to solve the issue faced with the momentum-based method. With the Nesterov accelerated momentum, we calculate gradients at the lookahead position, instead of calculating gradient at the current position. The lookahead position implies the position where the momentum would take us to.
Adaptive methods of gradient descent include the following:

1) Adagrad
2) Adadelta
3) RMSProp
4) Adam
5) Adamax
6) AMSGrad
7) Nadam
Analytical gradients are the gradients we calculated through backpropagation and the numerical gradients are the numerical approximation to the gradients.
Gradient descent is the most popular and widely used optimization algorithms used for training neural networks. Yes, Gradient descent is the first-order optimization method because with gradient descent we calculate only the first-order derivative.
Gradient descent is an optimization method used for training the network. First, we compute the derivatives of the loss function with respect to the weights of the network and then update the weights of the network using the below update rule:
  • Weight = weight - learning rate x derivatives
The matrix is often called the Jacobian matrix if it contains the first-order partial derivatives.
When the learning rate is small then we take a very small step and it slows down attaining the convergence and when the learning rate is large then we take a very large step and it may cause us to miss out on the global minimum.
  In gradient checking, first, we compute the analytical and approximated numerical gradients. Then we compare the analytical and numerical gradients. If they are not the same then there is an error with our implementation.

  we don’t have to check whether analytical and numerical gradients are exactly the same since the numerical gradient is just an approximation. So, we compute the difference between the analytical and numerical gradients and if their difference is very small say 1e-7 then our implementation is correct else we have a buggy implementation.
The convolutional neural network uses the special operation called convolution which is capable of extracting important features from the image. Since the convolutional operation extracts good features from the image, the accuracy of CNN is high compared to the other algorithms for the image data.
Select rollno, count (rollno) from Student
Group by rollno
Having count (rollno)>1
Order by count (rollno) desc;
PEMDAS Rule
Parenthesis
Exponentiation
Multiplication/Division
Addition
Subtraction
CNN is translation-invariant and this makes CNN more prone to misclassification. Say, for instance, we are performing a face recognition task then CNN checks only the presence of facial features such as eyes, nose, mouth, and ears. It will not check whether those features are present in the correct locations. If the images have all those features then it will be classified as the face irrespective of the location of the features. This is one of the major drawbacks of CNN.
The LeNet architecture consists of seven layers as given below:

a)Three convolutional layers
b)Two pooling layers
c)One fully connected layer
d)One output layer
We can break down a convolutional layer with a larger filter size into a stack of convolutional layers with smaller filter size and this is known as factorized convolution.
Suppose, we have a convolutional layer with a 5 x 5 filter then it can be broken down into two convolutional layers with 3 x 3 filters.
1x1 convolution implies that we use one filter of size 1 x 1. It is widely used for reducing the number of depth channels.
The inception network contains nine inception blocks. These nine inception blocks are stacked one above the other. First, we take the input image and we perform the convolutional operation with three filters of varying size which includes 1 x 1, 3 x3, and 5 x5. Then we feed the result of the convolutional operation to the next inception block.
Let’s suppose we are performing an object detection task. The object can appear anywhere in the image. That is, the object can be in the center region of the image, or it can be in the small corner of the image. Also, the shape of the object can vary from image to image. In some images, the object takes large shape while in other images the object takes small shape. Since the object in the image varies greatly in the image in terms of size and location, it is difficult to identify the object in the image if we use only a single filter with a fixed size. So in the inception network, we use multiple filters of varying sizes.
The architecture of VGG consists of convolutional layers followed by a pooling layer. VGG net uses 3 x 3 convolution and 2 x 2 pooling. It is often referred to as VGG-n and the n corresponds to a number of layers, excluding the pooling and softmax layer.
The different types of pooling include the following:

1) Max pooling
2) Average pooling
3) Sum pooling
Let’s say we performing an image classification task. First, we feed the image as input to the network and perform the convolution operation.  The convolution operation extracts the important features from the image and produces the feature map. Then we feed this feature map to the fully connected layer which performs the classification task.
The activation map which is obtained as a result of the convolutional operation will have a large dimension. To reduce the dimension of activation maps, we use the pooling layer.
When we pad the input matrix with zero values is called the same padding and instead of padding if we discard the region of input matrix which does not fit well with the filter matrix then it is called valid padding.
When we slide the filter matrix over the input matrix, in some cases, the filter matrix does not fit the input matrix perfectly, when the filter matrix does not fit the input matrix perfectly we apply padding.
If we set stride to a high value, then it takes us less time to compute but we might miss out on some important feature from the image. If we set stride to a low value, then we can learn the more detailed representation fo the image but it will take us a lot of time to compute.
In the convolution operation, we take the filter matrix and slide it over the input matrix by n number of pixels, perform element-wise multiplication, sum the result, and produce a single number. The number of pixels we choose to slide the filter matrix over the input matrix is often called the stride.
The matrix obtained as a result of convolution operation is often called activation maps or feature maps.
We take the input matrix and one more matrix called the filter matrix. We slide the filter matrix over the input matrix by n number of pixels, perform element-wise multiplication, sum up the results, and produce a single number and this operation is known as convolution.
The convolutional neural network uses the following three important layers:
Convolutional layer
Pooling layer
Fully connected layer
The convolutional neural network uses the special operation called convolution which is capable of extracting important features from the image. Since the convolutional operation extracts good features from the image, the accuracy of CNN is high compared to the other algorithms for the image data.
Connection Objects are used to provide a connection between Excel and Databases with the help of ADODB Objects. So, we can interact with the database and can use the SQL queries to fetch the data from the database. ADO stands for, ActiveX Data Objects.
VBA does not support all the OOPs concepts (VBA will support Polymorphism and Encapsulation, not supports Inheritance). Hence, VBA is called Object-Based Programming Language.
Conditional Formatting is used to format cells/Range based on a condition/Conditions. Ex: Highlighting a cell-based on cell Value
The slicer is used to filter the Table, Pivot Table data. Instead of using Filters section in a Pivot Table, we can use Slicer.
Goal Seek – is used to achieve your goal by changing the dependent value.
Ex: If you have taken a personal loan, and if you can able to pay the EMI of 6K instead of 10K, how many months do you need to close your personal loan?
UDF stands for User Defined Function, and these are custom functions. Using VBA, you can create your own functions and those can be used in Excel worksheets as normal built-in Functions.
Assigning a macro to a shape.
Assigning a macro to a button
Run a macro from the ribbon
Run a macro using a keyboard short cut key.
Using the TRIM () function, we can delete duplicate spaces and gives unique/single space between words.
Data Validation – is used to validate the Data in a cell/Range. In Data Validation, we have criteria such as List, Whole Number etc. And have custom criteria option where we can give function/formula.
There might be different ways to find duplicate values from a range. One of that is, using COUNTIF function we can find duplicate values.
Instead of VLOOKUP, we can use INDEX and MATCH functions. Limits of VLOOKUP is we cannot fetch the data from the left side of the Lookup range. Using INDEX-MATCH, we can fetch the data anyways.
Using Calculated Field
Creating a Table
Using OFFSET and COUNTA Functions
FORMULA – is a statement which written by the user for calculations. Ex: =1+2+3
FUNCTION – is a built-in formula by Excel. Ex: =SUM(1+2+3)
Slicer – is used to filter the Table, Pivot Table data visually.
Timeline – is used to filter the dates interactively by Year, Month, Quarter, and Day.
COUNT – Counts the number of cells that contain only numbers except for blank cells.
COUNTA – Counts the number of cells which contains alpha-numeric except blank cells.
Q9)Is it possible to make a single Pivot Table for multiple data sources? Yes, it is possible. Using the Pivot Table Data Modeling technique.
VLOOKUP – Using VLOOKUP, we can retrieve the data from left to right in the range/Table.
INDEX-MATCH – Using a combination of INDEX and MATCH, we can retrieve the data from left to right/right to left in a range/table.
Data > Get External Data section > Choose your data source
Option Explicit will force the user to declare variables. If the user uses undeclared variables, an error occurs while compiling the code.
Application – Workbooks – Worksheets – Range
A variant is the default data type in VBA.
Using the $ symbol.
Ex: $A $1 – Here Locked Column A and row 1
The function will return a value whereas Subroutine may or may not return a value. The function can be called in the procedure. We can create custom functions using FUNCTION like built-in functions.
No, it will not. VBA is an Object-based programming language, not Object Oriented Programming language.
ThisWorkbook – is the workbook where the VBA code is written.
ActiveWorkbook – is the workbook which is in Active state at present.
Step by step execution – F8
Breakpoints – F9
Using Debug.Print
Immediate Window
Watch Window
ADO –ActiveX Data Objects is a data access framework and is useful to get the data from the databases.
ODBC – Open Database Connectivity is useful to get the data from the external database.
OLEDB – Object Linking and Embedding, Database.
#N/A
#DIV/0!
#VALUE!
#REF!
#NAME
#NUM
Volatile functions recalculate the formula, again and again, so Excel workbook performance will be slow down. Volatile functions recalculate the formulas when any changes happen in the worksheet. Ex: NOW (), RAND ()
Dashboards: Dashboard is a visual display of the data and these are dynamic and live, so data is being updated in real-time and visuals can show changes from minute to minute.
Reports: Reports are not live and we use historical data to make reports. sometimes Reports are included with visuals such as Table, Graphs and Charts, Text, Numbers or anything.
Structured references – Instead of using cell references, we can use an Excel table name or the Column name for reference.
.XLS .XLSX .XLSM (Macro-enabled workbook) .XLSB (Binary format) .CSV (Comma Separated Values)
In 2 ways we can pass arguments to VBA
Functions
ByVal
ByRef
Code Module: Default module to write procedures.
User form: Helps to develop GUI (Graphical User Interface) applications.
Class Module: Allows to create new objects.
The Collection object contains a set of related groups of items as a single object.
Collections and Arrays are used to group variables. In Arrays, before using to start adding elements we normally set the size. But in Collection, we will not set the size, because we don’t know the number of elements in advance.
Sheet Module
Thisworkbook Module
Code Module
Userforms
Class Modules
Below are some tips –
Turn off screen updating
Declare the variables and avoid the “Variant” data type.
Disable events
Using the WITH statement.
Avoiding the Select statement
Select Case instead of If Then
Item
On Error Goto 0
On Error Resume Next
On Error Goto [Label]
Err Object
Err. Number
Err. Description of
Err. Source
Err. Raise
Error Function
Error Statement
Implicit Declaration: When variables are used without declaration is called Implicit declaration.
Explicit Declaration: Declaring variables before using is called Explicit Declaration.
Arithmetic Operators, Comparison Operators, Logical Operators etc.
There are 2 types of arrays available in VBA.
Single Dimensional Array: Single Dimensional array is used more often in the VBA. An array uses only one index. Multi-Dimensional Array: If an array has more than 1 dimension is called Multi-Dimensional Array.
CONCATENATE( ) – used to join several text strings to one string
TEXT( ) – Converting a value into text formatting
PROPER ( ) – Arranging the characters in a proper way.
LEFT ( ) – Returns the specified number of characters from the starting character.
I took a bottom-up approach.
Considering an ordinary, urban household with 4 individuals
Number of Maggi needed per month = 10
Therefore, per head consumption = (10/4) = 2.5 Maggi per person
Population = 1.3 billion
Urban population: 70% of total population
Above poverty line population: 40% of total population
Therefore, net population to consider:
1300*0.7*0.4 = 364 million. Population distribution: (Age-wise)
0 – 10 (consume less than 2.5 packets per month, say 2 packets): 20% of the population {which equals to (364*0.2*2) million packets per month = 145.6 million packets per month}
10 – 60 (consume 3 packets per month): 65% of the population {which equals to (364*0.65*3) million kg per month = 709.8 million packets per month}
60+ (consume less than 2.5 packets per month, 2 packets): 15% {which equals to (364*0.15*2) million packets per month = 109.2 million packets per month}
Total approximate consumption = (145.6 + 709.8+109.2) million packets/month = 964.6 million packets/month
Assuming a month of 30 days, per day consumption = (964.6/30) million packets per day = 32.15 million packets per day.
We can approach this problem in two ways:
Demand side
Supply side
I am going to solve using demand of t-shirts in the market
Total population of india : 1 bn (approx) Reach to internet : 40% =400 Mn
Reach of ecommerce companies to deliver products : 3/4th = 300Mn
Let's assume 50% are male and 50% are female
Lets solve for male population first: Now i have divided males in the four categories on the basis of age because demand demand of t-shirts for different age groups will be different
0–15 yr = 45 Mn, on an average, individual own 4 t shirts -> 4*45=180 Mn
16–22 yr = 23 Mn, on an average individual own 4 t shirts -> 4*23 = 92 Mn
23–50 yr = 65 Mn, on an average individual own 3 t shirts -> 3*65 = 195 Mn
50 - 80 yr = 18 Mn, on an average individual have 2 t shirts -> 2*18 = 36 Mn
Total t shirts own by men : 180 + 92+195+36 = 503 Mn ~ 500 Mn
Let's solve for female population now:
0–15 yr = 45 Mn, on an average individual own 2 t shirts -> 2*45=90 Mn
16–22 yr = 23 Mn, on an average individual own 4 t shirts -> 4*23 = 92 Mn
23–30 yr = 15 Mn, on an average individual own 3 t shirts -> 3*15 = 45 Mn
30 - 80 yr = 67 Mn -> we can neglect this section. Only a few ladies prefer to use t-shirts in this age group.
Total t-shirts own by females: 90+92+45 = 227 Mn ~ 230 Mn
Total t-shirts own by men + women = 500+230=730 Mn
Average life of a t-shirt = 2-year
Demand per year = 365 Mn ~ 360 Mn
Online portals provide coupons and offers but because of trust factor and fitting issues, people in India still prefer to buy offline, So I am assuming 30% of people buy a t-shirt from eCommerce portal and 70% are buying from market.
Total number of t-shirts sold through eCommerce platform per year in India= .3*360 = 108 Mn ~ 100 Mn per year
Number of t-shirts sold in India per day(From eCommerce portal) = 100 * 10^6/365 ~ 27,000
The population of India: 1200 mn Population above the poverty line: 70% 840 mn
Population below 14 years: 30%
Hence, proxy figure: 588 mn
Rural Population (70%): 410 mn
Rural Households: 82 Mn
Rural Mobile Penetration: Avg 2 per
household- 164 Mn
In rural areas assume that a new mobile is bought once in 3 years. Hence, new mobiles bought In the current year- 55 Mn Urban (30%):176 Mn
Assume Avg No of Mobiles per person: 1.5
Urban Mobile Penetration: 265 Mn
Assuming that a new mobile is bought once in 1.5 years. Hence new mobiles in the current year- 176 Mn
Total New Mobiles: 231 mn
Assuming 3 out of 10 new mobiles are smartphones
No. of smartphones sold=70 Mn
Observations:
35 million students enroll in India(Undergraduate, graduate, doctorate, diploma)
72% of 35 million graduate every year = 25 million
Students completing 10th grade = 20 million
Students completing 12th grade= 15 million
Unemployed graduates of the previous year= 15 million (Since 60% of 25 million graduates are unemployed)
GDP growth rate is 7%
Calculations:
40% of 25 million graduates are only employed= 10 million
Assuming 500,000 of the previous year’s graduates get a new job
100,000 starts working after 12th grade due to poverty, poor grades, etc
An estimate of 50,000 starts working after 10th grade due to poverty, poor grades, etc
10,000 people already on workforce end up with a new job
Total= 10 million + 500,000 + 100,000 + 50,000 + 10,000 = 10.66 million (approx)
Note:
Migrants working in India are negligible Due to urbanization, very few go for work without completing their 10th grade Increased feminism has a significant effect on the estimates
OYO Rooms Case Study
- The main KPIs for any online room booking company could be:-
a. Online Rating
b. Occupancy %
c. Average daily rate
d. Revenue per available room
e. Customer Satisfaction
f. Advertising ROI
- If I were to calculate, it should be equal to Total Revenue Per Room/Total rooms occupied
- Banners on websites
- Video ads on youtube
- Pamphlet distribution
- Hoarding
- Email/SMS
- The ALOS metrics make it easy to identify the length of stay of guests at your hotel. This is calculated by dividing the occupied rooms by a number of bookings. It is said that a higher number means an improved profit as less labor is required. On the other hand, lower ALOS results in reduced profit. The concept is that if a guest stays for a long period of time then it requires less labor. Whereas if several guests book rooms for one-nights for the same period of time then it requires more labor.
- To stay ahead of the competition you need to know how your hotel is performing in the local market. The MPI metrics can be used as a tool to compare your hotel’s market share with your competitors. It helps you to know how many guests are choosing your hotel as compared to other hotels in your location. It can be calculated by dividing your hotel’s occupancy by market occupancy and multiplying by 100. If the result is more than 100 that means you have a very good hold on the market. Else if it is less than 100 then it indicates your hotel isn’t performing well and losing a lot of bookings to your competitors.
The approach to such problems follows a MECE approach. MECE expands to Mutually Exclusive Collectively Exhaustive, which trivially means breaking your problem down to Non-overlapping segments which add up together to give your final solution.
Let's solve the guesstimate
The population of Delhi: 20 Mn
Children or college going = 20% of 20 Mn -> 4 Mn
Senior citizens = 20% of 20 Mn -> 4 Mn
Working people = 60% of 20 Mn -> 12 Mn
let there are 5 brands of car and each brand has 10 cars that are equally distributed. So in total, we have 50 models of cars running in the streets. This does not include luxury cars.br> Working-class people, let's assume half are married and half remain unmarried. So married -> 6 Mn and unmarried -> 6 Mn
Married couples:-
Number of married couples = 6 Mn/2 -> 3 Mn
I am assuming 10% belong to the rich class and prefer luxury cars and 20% cannot afford a car. The rest 70% has one car each.
70% of 3 Mn = 2.1 Mn
There is an equal distribution of above mentioned 50 cars among these 2.1 couples again. So the number of Swift Cars right now is 2.1 Mn / 50 = 0.042 Mn. I am assuming Swift car comes in 10 colors. Hence a number of red swift cars in married couples is 0.0042 Mn -> 42,000
Unmarried couples:-
Out of 6 Mn unmarried couples, Only 10% can afford mid-range nonluxury cars. Hence no of cars = 6 lakh. These are again divided into 50 models as above and each model has 10 colors. So number of red-colored swift cars among unmarried unmarried people = 6 lakh / 500 -> 12,000
Senior citizens
Out of 2 Mn families(4 Mn people), 20% i.e. 0.4 Mn families own a car. Again, as above, these cars are divided into 50 models with each model having 10 colors. So 4 lakh/500 -> 8,000
Total number of red-colored swift cars in Delhi = 42,000 + 12,000 + 8,000 -> 62,000
Possible answer:
The candidate would begin by discussing the market dynamics in Brazil as well as globally on which he/she has to base the suggestion. Furthermore, an idea is to be framed up about the cost, market, value, customers, transportation facility, and price if the steel is to be exported. Also, Brazil has some taxes on foreign goods export which would only add up to the price. Since the local market is more profitable than international trade, it is advisable to try out the products first in the local market of Brazil since there is a chance of price war.
Vectors can be combined from 2 to 1 by using the c() function Example.
> first <- c(1,2,3,4)
> second <- (“a”, “b”, “c”)
> third <- c(first, second)
> print(third)
[1] “1” “2” “3” “4” “a” “b” “c”
The matrix can have only 2 dimensions whereas an array can have as many dimensions as you want. Matrix is defined with the help of data, number of rows, number of columns and whether the elements are to be put in row-wise or column-wise.
In array, you need to give the dimension of the array. An array can be of any number of dimensions and each dimension is a matrix. For example, a 3x3x2 array represents 2 matrices each of dimension 3x3.
A data frame can contain vectors with different inputs and a matrix cannot. (You can have a data frame of characters, integers, and even other data frames, but you can't do that with a matrix. A matrix must be all the same type.)
So, the data frame can have a different vector of character, numbers, logic, etc. and it is still cool. But, for a matrix, you need only one type of data type.
General format is >Temp_matrix< – matrix (vector, nrow=r ,ncol=c , byrow=FALSE, dimnames = list ( char_vector_ rowname, char_vector_colnames))
The repeat loop executes a sequence of statements multiple times. It doesn’t put the condition at the same place where we put the keyword repeat. Example
> name <- c(“Pappu”, “John”)
> temp <- 5
> repeat {print(name)
temp <- temp+2
if(temp > 11) {
break
}
}
So, this will return the name vector 4 times. First, it prints the name and then increase the temp to 7 and so on.
The sequence-to-sequence model (seq2seq) is the many-to-many RNN architecture. It is widely used in various applications where we need to map an arbitrary-length input sequence to an arbitrary-length output sequence. The example includes music generation, chatbots, and more.
Unlike RNN, bidirectional RNN is used for reading the inputs in both directions. It consists of two different layers of hidden units. In one layer, the hidden states are shared from left to right, and in the other layer, they are shared from right to left and both of these layers connect from the input layer to the output layer.
One problem with LSTM is that it involves too many parameters. This is mainly due to the presence of many gates and states in the LSTM cell. Since LSTM has too many parameters, it increases the training time. So, to avoid this, we use gated recurrent units which is just a simplified and modified version of LSTM.
In LSTM, the cell state is mainly used for storing the information and it is also referred to as the internal memory whereas the hidden state is mainly used for computing the output.
LSTM differs from RNN by introducing three special gates called input gate, forget gate and output gate.
The recurrent network is preferred over feedforward networks when we want to perform a sequential task. Since the recurrent neural network store the past information in the hidden state, the recurrent neural network is very effective for a sequential task than the feedforward network.
We can prevent the exploding gradient problem by using gradient clipping. With the gradient clipping method, we normalize the gradients according to an L2 norm and clip the gradient value to a specific range.
We can prevent the vanishing gradient problem by using the ReLu activation function instead of tanh or sigmoid activation. We can also avoid vanishing gradient problem by using a variant of RNN called LSTM.
Suppose, we initialize the weights of the network randomly with small values. During backpropagation, we compute the derivative of the hidden layer and multiply them by weights at every step while moving backward.
This derivative and weights, both of which are a small number. When we multiply two numbers which are small then the result will be a smaller number. So, when we multiply the weights and derivative at every step then our gradient becomes an infinitesimally small number and this is called vanishing gradient problem.
Since the RNN uses its hidden state as a memory for storing past information, it is widely useful in many sequential tasks such as text generation, time series prediction, and so on.
In the recurrent neural network, to predict the output, instead of using only the current input we also use the previous hidden state. The previous hidden state holds the information about what the network has seen so far whereas in the feedforward network, to predict the output, we use only the current input.
In the while loop the condition is tested and the control goes into the body only when the condition is true
Example
> name <- c(“Pappu”, “John”)
> temp <- 5
> repeat (temp<11){
print(name)
temp <- temp+2
}
The for loop is not limited to integers. You can pass character vectors, logical vectors, lists, or expressions. Example.
>x<- LETTERS[1:2]
for ( i in x) {
print(i)
}
[1] “A”
[2] “B”
Name_of_function<- function(argument_1,argument_2,..) {
function body
}
The argument is the place holder, whenever a function is invoked, it passes a value to the argument. Arguments are optional
Elements in a vector can be sorted using the function sort()
Example.
>temp <- c(3,5,2,6,7,1)
>sort_temp<- sort(temp)
>print(sort_temp)
[1] 1 2 3 5 6 7
>rev_sort<- sort(temp, decreasing = TRUE)
[1] 7,6,5,3,2,1
This function also works with the words
In R, subset() functions help you to select variables and observations while through sample() function you can choose a random sample of size n from a dataset.
**Suppose there is  adata.frame
data_frame_example<- data.frame(a = c(10, 20, 30), b = c(40, 50, 60), c(70, 80, 90))...
The global variable b has a value of 4. The function f has an argument 2 and the function’s body has the local variable b with the value 3. So function f(2) will return 3^3 + g(2) and g(2) will give the value 2*4 = 8 where 4 is the value of b.
Thus, the answer is 35
max_salary_person<- subset(data, salary == max(salary))
print(max_salary_person)
temp 300000 &as.Date(DOJ) >as.Date(“2016-01-01”))
write.csv(temp,”output.csv”,row.names = FALSE)
new_temp<- read.csv(“output.csv”)
print(new_temp)
Example.
a<- c(1,2,3,4)
b<- c(“Amit”,”Sumit”,”Gaurav”)
c<- c(“TCS”,”CTS”,”Musigma”)
df<- cbind(a,b,c)
print(df)
We can merge two data frames by using the merge() function. The data frames must have same column names on which the merging happens.
Example.
df1<- data.frame(id<- c(1:6), name <- c(rep(“Amit”,3), rep(“Sumit”,3))
df2<- data.frame(id<-c(7,8,9), name<- c(rep(“Nitin”,2), rep(“Paplu”,1))
*outer join
merge(x=df1, y=df2, by = “id”, all = TRUE)
This all = TRUE will give you the outer join, so the new data set will have all the value from both the data frame merged on the id
The  CycleGAN maps the data from one domain to another domain. That is, in the CycleGAN, we map the distribution of images from one domain to the distribution of images in another domain.
As mentioned in the tensorflow documentation TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments. TensorFlow Serving makes it easy to deploy new algorithms and experiments while keeping the same server architecture and APIs. TensorFlow Serving provides out-of-the-box integration with TensorFlow models but can be easily extended to serve other types of models and data.
Eager execution in TensorFlow allows for rapid prototyping. The eager execution follows the imperative programming paradigm, where any operations can be performed immediately, without having to create a graph.
TensorBoard is TensorFlow's visualization tool used to visualize a computational graph. It can also be used to plot various quantitative metrics and the results of several intermediate calculations.
The feed_dict parameter is the dictionary where the key represents the name of the placeholder and the value represents the value of the placeholder.
We use the variables for storing the values. Variables are used as input to several other operations in a computational graph.
We can think of placeholders as variables, where we only define the type and dimension, but do not assign the value. Values for the placeholders will be fed at runtime.
TensorFlow sessions can be created as tf.Session()
In order to run the data flow graph, we use a TensorFlow session.
Every computation in TensorFlow is represented by a graph called data flow graph or computational graph. It consists of several nodes and edges, where nodes are mathematical operations, such as addition and multiplication, and edges are tensors.
A tensor is a multidimensional array.
Being the rapidly evolving field, there are new improvements and advancements being published every day. So make sure to answer questions regarding the different variants of BERT, ELMo, XLnet, and so on.
We can evaluate embedding by performing word similarity, projecting them to the embedding space, visualization, and clustering.
In the CBOW model, the goal of the network is to predict a target word given its surrounding words whereas, in the skip-gram model, the goal of the network is to predict surrounding words given a target word.
The word embeddings capture the syntactic and semantic meanings of a word which helps our network to understand the word better.
Word embeddings are the vector representations of words in a vector space.
The variational autoencoders are the generative model unlike other types of autoencoders. Just like GAN, variational autoencoders learn the distribution of the training set and they are widely used in the generative tasks.
In order to make sure that our encodings are more robust to small perturbations present in the training set, we use the contractive autoencoders.
The contractive autoencoders use a new penalty term in the loss function which penalizes the representations that are too sensitive to the input.
The sparse autoencoder introduces a special constraint in the loss function called a sparse constraint. The sparse constraint is used to make sure that the autoencoder is not overfitting to the training data when we set the many nodes in the hidden layer.
When we set many nodes in the hidden layer then we can learn a better and robust latent representation of the input. But the problem is when we keep more nodes in the hidden layer, then the autoencoders overfit the training data.
To combat this problem of overfitting, we need a sparse autoencoder.
We can use the denoising autoencoders for denoising images. First, we corrupt the input by adding some noise and feed this corrupted input to the encoder instead of feeding the raw input.
While learning the representation of the input, the encoder will learn that the noise is unwanted information and removes its representation. Thus, the encoder learns the compact representation of the input without including the noise and by keeping only necessary the information and map the learned representation to the bottleneck.
Next, the decoder takes the bottleneck created by the input and reconstructs the image. Since the bottleneck does not contain any representation of the noise, the decoder can generate a denoised image from the bottleneck.
We feed the input image to the encoder which consists of a convolutional layer. The convolutional layer performs the convolution operation and extracts important features from the image.
Next, we apply the max pooling operation to retain only the important features of the image and obtain a latent representation of the image, called a bottleneck.
To the decoder, we feed the bottleneck as an input. The decoder consists of deconvolutional layers and it performs the deconvolution operation and tries to reconstruct the image from the bottleneck.
When the code or latent representation has the dimension higher than the dimension of the input then the autoencoder is called the overcomplete autoencoder. On the contrary, when the code or latent representation has the dimension lower than the dimension of the input then the autoencoder is called the undercomplete autoencoder.
The encoder takes the given input and outputs the low dimensional latent representation of the input. The decoder takes this low dimensional latent representation generated by the encoder as an input and tries to reconstruct the original input.
The autoencoders map the data of a high dimension data to a low-level representation. This low-level representation of data is called as latent representation or bottleneck. The bottleneck comprises of only meaningful and important features that represent the input.
The difference between the autoencoder and PCA is that PCA uses linear transformation for dimensionality reduction while the autoencoder uses a nonlinear transformation for dimensionality reduction.
You must be wondering that it’s very easy to use a CSV file by putting the name inside the read.csv() function. But, in most of the cases, we also need to put some extra conditions in order to get things right and less frustrating for us.
Use the below syntax for the use.
my_data<- read.csv(“filename.csv”, stringsAsfactors = FALSE,  strip.white=TRUE, na.strings=c(“NA”,””))
stringsAsFactors = FALSE tells R to keep
character variables as they are rather than convert to factors.
strip.white = TRUE removes spaces at the start and end of character elements. R treats “game” and ” game” differently, which is not usually desired.
na.strings = c("NA","") tells R that in addition to the usual NA, empty strings in columns of character data are also to be treated as missing
lapply() function is used when you want to apply a function to each element of a list in turn and get a list back.
Example.
x<- list(a=1, b=1:3, c=10:100)
lapply(x,FUN=length)
$a
[1]1
$b
[1]3 $c
[1]91
You can use other functions like max, min, sum, etc.
supply() function is used when you want to apply a function to each element of a list in turn, but you want a vector back, rather than a list.
Vector is useful sometimes because it will get you a set of values and you can easily perform an operation on it.
Example.
x <-list(a =1, b =1:3, c =10:100)
#Compare with above; a named vector, not a list
sapply(x, FUN = length)
a b c
1391 sapply(x, FUN = sum)
a b c
165005
A scatterplot is a graph which shows many points plotted in the Cartesian plane. Each point holds 2 values that are present on the x and y-axis. The simple scatterplot is plotted using plot() function.
The syntax for scatterplot is:-
plot(x, y ,main, xlab, ylab, xlim, ylim, axes)
Where
x is the data set whose values are the horizontal coordinates
y is the data set whose values are the vertical coordinates main is the tile in the graph xlab and ylab is the label in the horizontal and vertical axis xlim and ylim are the limits of values of x and y used in the plotting axes indicate whether both axes should be there on the plot
plot(x =input$wt,y=input$mpg,
xlab="Weight",
ylab="Milage",
xlim= c(2.5,5),
ylim= c(15,30),​
main="Weight vsMilage")
timer<- function(time) { print(time) while(time!=0) { Sys.sleep(1) time<- time - 1 print(time) } }
countdown(5)
[1] 5
[2] 4
[3] 3
[4] 2
[5] 1
Every matrix can be an array but every array need not be a matrix. A matrix cannot have more than 2 dimensions, whereas an array can be multi dimensional.
lucky_number = {‘Amit’:4,’Rahul’:6,’Nihar’:8}
for name,number in lucky_number.items():
  ​print(name+’prefers’+str(number))
lucky_number = {‘Amit’:4,’Rahul’:6,’Nihar’:8}
for name in lucky_number.keys():
print(name)
filename = 'abc.txt'
with open(filename) as file_object:
  ​lines = file_object.readlines()
for line in lines:
  ​print(line)
x = “What is your age?”
inp = input(x)
try:
​inp = int(inp)
except ValueError:
  ​print(“Sorry, Please Try again latter”)
else:
  ​print(“That’s a beautiful age “)
i = 2
while(i < 20):
j = 2
while(j i/j) :
print (i," is a prime number")
i = i + 1
list_example = ['Amit','Sumit','Rahul']
print(list_example)
list_example[1] = 'Kamal'
print(list_example)
['Amit', 'Sumit', 'Rahul']
['Amit', 'Kamal', 'Rahul']
def game_name(name):
for x in game_name
print(x)
example = [‘Cricket’,’Football’,’TT’]
game_name(example)
def pizza(size, *toppings):
  print("\nMaking a " + size + " pizza.")
  print("Toppings:")
  for topping in toppings:
     print("- " + topping)
# Make three pizzas with different toppings.
make_pizza('small', 'pepperoni')
make_pizza('large', 'bacon bits', 'pineapple')
make_pizza('medium', 'mushrooms', 'peppers', 'onions', 'extra cheese')
Splitting a dataset into a train and test is one of the initial stages of most of the machine learning models. Following is how you can split the dataset in python:-
from sklearn.model_selection import train_test_split
X_train,X_test,Y_train,Y_test = train_test_split(dataframe_name,target_variable,test_size=0.3, random_state=42)
dataframe_name = the complete dataset as a panda dataframe target_variable = the name of the target variable test_size = 0.3 denotes 70-30 split of the dataset in train and test random_state = 42, Look for the explanation in the next question
It will print Avatto four times str='Avatto' print(str*4) AvattoAvattoAvattoAvatto
import matplotlib.pyplot as plt from numpy import cos x = [x*0.01 for x in range(100)] y = cos(x) plt.plot(x,y) plt.show()
The recession which happened in the past, and it is supposed to happen in the coming few years based on cyclicity
- Trend, Seasonality, and Cyclicity
- When a particular behavior is repeated time and again. Generally we talk about a span of 3 to 10-12 years.
There are many ways to improve the performance of a Python program. Some of these are as follows:
Data Structure: We have to select the right data structure for our purpose in a Python program.
Standard Library: Wherever possible, we should use methods from the standard library. Methods implemented in the standard libraries have much better performance than user implementation.
Abstraction: At times, a lot of abstraction and indirection can cause slow performance of a program. We should remove the redundant abstraction in code.
Algorithm: The use of the right algorithm can make a big difference in a program. We have to find and select a suitable algorithm to solve our problem with high performance.
Python is strong that even Google uses it. Some of the benefits of using Python are as follows:
Efficient: Python is very efficient in memory management. For a large data set like Big Data, it is much easier to program in Python.
Faster: Though Python code is interpreted, still Python has a very fast performance.
Wide usage: Python is widely used among different organizations for different projects. Due to this wide usage, there are thousands of add-ons available for use with Python.
Easy to learn: Python is quite easy to learn. This is the biggest benefit of using Python. Complex tasks can be very easily implemented in Python.
By default, every source code file in Python is in UTF-8 encoding. But we can also specify our own encoding for source files. This can be done by adding the following line after #! line in the source file.
# -*- coding: encoding -*-
In the above line, we can replace encoding with the encoding that we want to use.
PEP 8 is a style guide for Python code. This document provides the coding conventions for writing code in Python. Coding conventions are about indentation, formatting, tabs, maximum line length, import organization, line spacing, etc. We use PEP 8 to bring consistency to our code. We consistency it is easier for other developers to read the code.
Pickling is a process by which a Python object hierarchy can be converted into a byte stream. The reverse operation of Pickling is Unpickling.
Python has a module named pickle. This module has the implementation of a powerful algorithm for serialization and de-serialization of Python object structure.
Some people also call Pickling as Serialization or Marshalling.
With Serialization, we can transfer Python objects over the network. It is also used in persisting the state of a Python object. We can write it to a file or a database.
There is a private heap space in Python that contains all the Python objects and data structures. In CPython there is a memory manager responsible for managing the heap space.
There are different components in Python memory manager that handle segmentation, sharing, caching, memory pre-allocation, etc.
Python memory manager also takes care of garbage collection by using Reference counting algorithm.
We can use Static Analysis tool called PyChecker for this purpose.

PyChecker can detect errors in Python code. PyChecker also gives warnings for any style issues.

Some other tools to find bugs in Python code are pylint and pyflakes.
In Python, Tuple and List are built-in data structures. Some of the differences between Tuple and List are as follows:
1) Syntax: A Tuple is enclosed in parentheses: E.g. myTuple  = (10, 20, “apple”);
A List is enclosed in brackets: E.g. myList = [10, 20, 30];
2) Mutable: Tuple is an immutable data structure. Whereas, a List is a mutable data structure.
3) Size: A Tuple takes much lesser space than a List in Python.
4) Performance: Tuple is faster than a List in Python. So it gives us a good performance.
5) Use case: Since Tuple is immutable, we can use it in cases like Dictionary creation. Whereas, a List is preferred in the use case where data can alter.
A Python Decorator is a mechanism to wrap a Python function and modify its behavior by adding more functionality to it. We can use @ symbol to call a Python Decorator function.
Every argument in a Python method is an Object. All the variables in Python have reference to an Object. Therefore arguments in Python method are passed by Reference.
Since some of the objects passed as reference are mutable, we can change those objects in a method. But for an Immutable object like String, any change done within a method is not reflected outside.
In SQL, NVL() converts a null value to an actual value. Data types that can be used are date, character, and number. The data type must match with each other i.e. expr1 and expr2 must of the same data type. NVL (expr1, expr2) expr1 is the source value or expression that may contain a null. expr2 is the target value for converting the null. In SQL, NVL() converts a null value to an actual value. Data types that can be used are date, character, and number. The data type must match with each other i.e. expr1 and expr2 must of the same data type. NVL (expr1, expr2) expr1 is the source value or expression that may contain a null. expr2 is the target value for converting the null.
The COALESCE() function examines the first expression if the first expression is not null, it returns that expression; Otherwise, it does a COALESCE of the remaining expressions.
The advantage of the COALESCE() function over the NVL() function is that the COALESCE function can take multiple alternate values. In simple words COALESCE() function returns the first non-null expression in the list.
EmpNo EName Job MGR HireDate Sal Comm DeotNo
1234 Amit Waiter 8382 19-Oct-18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT table_name, num_rows
FROM user_tables
WHERE table_name=’Employee’;
EmpNo EName Job MGR HireDate Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct-18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
The FETCH argument is used to return a set of a number of rows. FETCH can’t be used itself, it is used in conjunction with OFFSET.
SELECT column_name(s)
FROM table_name
ORDER BY column_name
OFFSET rows_to_skip
FETCH NEXT number_of_rows ROWS ONLY;
DeptNo Dname Loc
50 Service Delhi
51 Account Mumbai
SELECT Department_Name,
Max(Salary)
FROM Department_Table
GROUP BY Department_Name
Though the variable PaymentYear has already been defined in the first line of the query, but this is not the correct logical process order. The correct query will be SELECT Id, Year(PaymentDate) as PaymentYear FROM Bill_Table WHERE Year(PaymentDate)>2018;
The order of query goes like this:-
FROM – Choose and join tables to get the raw data
WHERE – First filtering condition
GROUP BY – Aggregates the base data
HAVING – Apply condition on the base data
SELECT – Return the final data
ORDER BY – Sort the final data
LIMIT – Apply a limit to the returned data
It assigns a unique id to each row returned from the query ,even if the ids are the same.
Sample query:- SELECT emp.*, row_number() over (order by salary DESC) Row_Number from Employee emp;

Employee Name Salary Row_Number
Amit 7000 1
Bhargav 6000 2
Chirag 6000 3
Dinesh 5000 4
Esha 3000 5
Farhan 3000 6

Even when the salary is the same for Bhargav and Chirag, they have a different Row_Number, this means that the function row_number just gives a number to every row
The RANK() function is used to give a rank and not a row number to the data set. The basic difference between RANK() and ROW_NUMBER is that Rank will give equal number/rank to the data points with the same value. In the above case, RANK() will give a value of 2 to both Bhargav and Chirag and thus will rank Dinesh as 4. Similarly, it will give rank 5 to both Esha and Farhan.
Employee Name Salary Row_Number
Amit 7000 1
Bhargav 6000 2
Chirag 6000 2
Dinesh 5000 4
Esha 3000 5
Farhan 3000 6
SELECT emp.*, RANK() over (order by salary DESC) Ranking from Employee emp;
NTILE() function distributes the rows in an ordered partition into a specific number of groups. These groups are numbered. For example, NTILE(5) will divide a result set of 10 records into 5 groups with 2 records per group. If the number of records is not divided equally in the given group, the function will set more records to the starting groups and less to the following groups.
Employee Name Salary Row_Number
Amit 7000 1
Bhargav 6000 1
Chirag 6000 2
Dinesh 5000 2
Esha 3000 3
Farhan 3000 3
SELECT emp.*, NTILE(3) over (order by salary DESC) as GeneratedRank from Employee emp This will divide the complete data set in 3 groups from the top. So the GeneratedRank will be 1 for Amit and Bhargav, 2 for Chirag and Dinesh: 3 for Esha and Farhan
This gives the rank of each row within a result set partition, with no gaps in the ranking values. Basically there is no gap, so if the top 2 employees have the same salary then they will get the same rank i.e. 1, much like the RANK() function. But, the third person will get a rank of 2 in DENSE_RANK as there is no gap in the ranking whereas the third person will get a rank of 3 when we use the RANK() function. The syntax below:-
Employee Name Salary Row_Number
Amit 7000 1
Bhargav 6000 2
Chirag 6000 2
Dinesh 5000 3
Esha 3000 4
Farhan 3000 5
SELECT emp.*, DENSE_RANK() OVER (order by salary DESC) DenseRank from Employee emp;
Employee Name Salary Row_Number
Amit 7000 1
Bhargav 6000 2
Chirag 6000 3
Dinesh 5000 4
Esha 3000 5
Farhan 3000 6
SELECT EmpID, EmpName
FROM Employee
where EmpName like ‘[aeiou]%’
Employee Name Salary Row_Number
Amit 7000 1
Bhargav 6000 2
Chirag 6000 3
Dinesh 5000 4
Esha 3000 5
Farhan 3000 6
SELECT EmpID,EmpName
FROM Employee
where EmpName like ‘[aeiou]%[aeiou]’
There are three types of statements in SQL:-
a. DDL – Data Definition Language
b. DML – Data Manipulation Language
c. DCL – Data Control Language
It is used to define the database structure such as tables. It includes 3 commands:-
a. Create – Create is for creating tables to CREATE TABLE table_name ( column1 data type, column2 data type, column3 datatype,....
);
b. Alter – Alter table is used to modifying the existing existing table object in the database. ALTER TABLE table_name
ADD column_name datatype
c. Drop – If you drop a table, all the rows in the table are deleted and the table structure is removed from the database. Once the table is dropped, you can’t get it back
Data Manipulation Language is used to manipulate the data in records. Commonly used DML commands are Update, Insert and Delete. Sometimes SELECT command is also referred as a Data Manipulation Language.
Data Control Language is used to control the privileges by granting and revoking database access permission
a. AVG()
b. COUNT()
c. MAX()
d. MIN()
e. SUM()
Normalization is used to organize the data in such a manner that data redundancy will never occur in the database and avoid insert, update, and delete anomalies.
There are 5 forms of Normalization
First Normal Form (1NF): It removes all duplicate columns from the table. Creates a table for related data and identifies unique column values
Second Normal Form (2NF): Follows 1NF and creates and places data subsets in an individual table and defines the relationship between tables using a primary key
Third Normal Form (3NF): Follows 2NF and removes those columns which are not related through primary key
Fourth Normal Form (4NF): Follows 3NF and does not define multi-valued dependencies. 4NF also known as BCNF
SELECT * FROM Employee INTERSECT SELECT * FROM Employee1
A. Full Outer Join is a combination of Left Outer and Right Outer Join in SQL
A. Right Outer Join is similar to Left Outer Join in SQL
The OFFSET argument is used to identify the starting point to return rows from a result set. Basically, it excludes the first set of records.
OFFSET can only be used with the ORDER BY clause. It cannot be used on its own. OFFSET value must be greater than or equal to zero. It cannot be negative, else return an error. SELECT column_name(s)
FROM table_name
WHERE condition
ORDER BY column_name
OFFSET rows_to_skip ROWS;
DROP is a Data Definition Language (DDL) command which removes the named elements of the schema like relations, domains or constraints and you can also remove an entire schema using the DROP command.
DROP SCHEMA schema_name RESTRICT;
DROP Table table_name CASCADE;
DELETE is a Data Manipulation Language (DDL) command and used when you want to remove some or all the tuples from a relation. If WHERE clause is used along with the DELETE command it removes only those tuples which satisfy the WHERE clause condition but if WHERE clause is missing from the DELETE statement then by default all the tuples present in relation are removed.
DELETE FROM relation_name
WHERE condition;
There were some output related questions where a table was given, mostly on the group by, order by, top, etc. command
Gradient checking is used for debugging the gradient descent algorithm and to make sure that we have a correct implementation.

  That is, when we implement the gradient descent method for the complex neural network, even with buggy implementations, the network will learn something.

  But the buggy implementation will not be as optimal as a bug-free implementation. So to ensure that we have the bug free implementation of gradient descent we perform gradient checking.
A function is called a convex function when it has only one minimum value and a function is called a non-convex function when it has more than one minimum value.
With gradient descent, we update the parameters of the model only after iterating through all data points present in our training set. Let’s say we have 10 million data points. Now, even to perform a single parameter update, we have to iterate through all the 10 million data points and then we update the parameter of the network. This is will be a very time-consuming task and takes us a lot of training time. So, to combat this drawback of gradient descent, we can stochastic gradient descent.
With stochastic gradient descent, we don’t have to update the parameters of the network only after iterating through all data points present in our training set. Instead, we can update the parameter of the network after iterating through every single point in the training set.
   With the mini-batch gradient descent, we don’t update the parameters of the network after iterating through every single data point in our training set. Instead, we update the parameters of the network after iterating through some n number of data points.

   Say n is 32, then it implies that we update the parameter of the network after iterating through every 32 data points in our training set.
One problem we face with SGD and mini-batch gradient descent is that there will be too many oscillations in the gradient steps. This oscillation happens because we update the parameter of the network after iterating through every point or every n data points and thus the direction of the update will possess some variances causing oscillation in the gradient steps.
This oscillation leads to slow training time and makes it's hard to reach the convergence. To avoid this issue we use momentum-based gradient descent.
pet_name = {'Nitin':['Kamal','Chintu'],
'Richa':['Shankar','Megha']}
for name,pet in pet_name.items():
print(name)
for x in pet:
print('-',x)
Gradient descent - Update the parameter of the network after iterating through all the data points present in the training set.
Stochastic gradient descent - Update the parameter of the network after iterating through every single data points present in the training set.
Mini-batch gradient descent - Update the parameter of the network after iterating through some n number of data points present in the training set.
In Python, we can use dict data type to store key value pairs. In this example, customer name can be the key and their location can be the value in a dict data type.
Dictionary is an efficient way to store data that can be looked up based on a key.
Output of this will be John Smith This is an example of Slicing. Since we are slicing at the same index, the first name[:5] gives the substring name upto 5th location excluding 5th location. The name[5:] gives the rest of the substring of name from the 5th location. So we get the full name as output.
The output of this code is following:
[]
Even though the list has only 2 elements, the call to thelist with index 3 does not give any index.
Some of the popular libraries of Python used for Data analysis are:
Pandas: Powerful Python Data Analysis Toolkit
SciKit: This is a machine learning library in Python.
Seaborn: This is a statistical data visualization library in Python.
SciPy: This is an open source system for science, mathematics and engineering implemented in Python.
To execute a Python script in Unix, we need to have Python executor in Unix environment.
In addition to that we have to add following line as the first line in a Python script file.
#!/usr/local/bin/python
This will tell Unix to use Python interpreter to execute the script.
In Python, enumerate() function is an improvement over regular iteration. The enumerate() function returns an iterator that gives (0, item[0]).
E.g. >>> thelist=['a','b']
>>> for i,j in enumerate(thelist): ...
print i,j
...
0 a
1 b
In Functional Programming, we decompose a program into functions. These functions take input and after processing give an output. The function does not maintain any state.
Python provides built-in functions that can be used for Functional programming. Some of these functions are:
Map()
reduce()
filter()
Event iterators and generators can be used for Functional programming in Python.
We can create a common module with variables that we want to share.
This common module can be imported in all the modules in which we want to share the variables.
In this way, all the shared variables will be in one module and available for sharing with any new module as well.
We use ‘is’ to check an object against its identity.
We use ‘==’ to check equality of two objects.
E.g.
>>> lst = [10,20, 20]
>>> lst == lst[:]
True
>>> lst is lst[:]
False
Python provides a profiler called cProfile that can be used for profiling Python code.
We can call it from our code as well as from the interpreter.
It gives use the number of function calls as well as the total time taken to run the script.
We can even write the profile results to a file instead of standard out.
In Python, we can use the debugger pdb for debugging the code. To start debugging we have to enter following lines on the top of a Python script.
import pdb
pdb.set_trace()
After adding these lines, our code runs in debug mode. Now we can use commands like breakpoint, step through, step into etc for debugging.
Python provides a useful method issubclass(a,b) to check whether class a is a subclass of b.
E.g. int is not a subclass of long
>>> issubclass(int,long)
False
bool is a subclass of int
>>> issubclass(bool,int)
True
Both split() function and slicing work on a String object. By using split() function, we can get the list of words from a String.
E.g. 'a  b c '.split() returns [‘a’, ‘b’, ‘c’]
Slicing is a way of getting substring from a String. It returns another String.
E.g. >>> 'a b c'[2:3] returns b
We can implement exception handling to handle error conditions in Python code. If we are expecting an error condition that we cannot handle, we can raise an error with appropriate message.
E.g. >>> if student_score < 0: raise ValueError(“Score can not be negative”)

If we do not want to stop the program, we can just catch the error condition, print a message and continue with our program.
E.g. In following code snippet we are catching the error and continuing with the default value of age.

#!/usr/bin/python
try:
age=18+'duration'
except:
print("duration has to be a number")
age=18
print(age) Powerhouse, Knowledge. Top 50 Python Interview Questions and Answers . Kindle Edition.
In Python, we get a built-in sequence called list. We can call standard functions like append() and extend() on a list.
We call append() method to add an item to the end of a list.
We call extend() method to add another list to the end of a list.
In append() we have to add items one by one. But in extend() multiple items from another list can be added at the same time.
To retrieve data from a database we have to make use of the module available for that database. For MySQL database, we import MySQLdb module in our Python script.
We have to first connect to a specific database by passing URL, username, password and the name of database.
Once we establish the connection, we can open a cursor with cursor() function. On an open cursor, we can run fetch() function to execute queries and retrieve data from the database tables.
We can do so using building a recommendation engine. The easiest we can do is to show content that is popular with other users, which is still a valid strategy if, for example, the contents are news articles. To be more accurate, we can build a content-based filtering or collaborative filtering. If there’s enough user usage data, we can try collaborative filtering and recommend contents other similar users have consumed. If there isn’t, we can recommend similar items based on the vectorization of items (content-based filtering).
Build a master dataset with local demographic information available for each location.
-local income levels
-proximity to traffic
-weather
-population density
-proximity to other businesses
-a reference dataset on local, regional, and national macroeconomic conditions (e.g. unemployment, inflation, prime interest rate, etc.)
-Any data on the local franchise owner-operators, to the degree the manager
-Identify a set of KPIs acceptable to the management that had requested the analysis concerning the most desirable factors surrounding a franchise. Quarterly operating profit, ROI, EVA, pay-down rate, etc.
-Run econometric models to understand the relative significance of each variable
-Run machine learning algorithms to predict the performance of each location candidate
-Based on the past pickup location of passengers around the same time of the day, day of the week (month, year), construct a travel map -Based on the number of past pickups -Account for periodicity (seasonal, monthly, weekly, daily, hourly) -Special events (concerts, festivals, etc.) from tweets
The following points were discussed:- a. Find out the place where people have mostly searched for 5 or 7-star hotels b. Find the place where the average annual income is high, maybe Bangalore, Pune, Delhi, Hyderabad, etc. c. Look for that place which is known for tourism as it will attract foreign customers d. Look for that area which has good facilities around like popular restaurants, pubs, malls, etc. e. Look for that city where there are all the necessary facilities like airport near the city, railway station, etc. f. Look for that city where you can get good service from third party vendors for basic services like laundry, service employees, security service, etc.
Height of all the employees on this floor or in this office
Honestly, I did not expect such a topic in a case study. I took some 4-5 minutes to shape my idea. Following are the points on which we discussed:- 1. TVF has some 10Million subscriber on Youtube, and it releases its video on Youtube after a week of its original release on the TVF website. These videos give it a good amount of money to keep the show running 2. The main reason for TVF to move to its own website was to create an ecosystem comparable to Netflix so that people buy a subscription to watch the show. 3. Netflix charges some $9 for subscription, TVF could be planning to launch it’s series exclusively to any of these and can get some part of the subscription. Even a dollar per person can get them close to 10Million dollars 4. The estimated revenue of a Youtube channel with 10 Million subscribers is ~500,000 dollars per year. 5. Apart from these, a major chunk of the production cost is taken care by the sponsor of the show. For example Tiago in Trippling, Kingfisher in Pitchers, etc. So the production cost is next to zero for the episodes 6. TVF is also going for its own website and raising funding to acquire customers and drive them to their website It’s hard to get a $10 subscription, but even a basic subscription or tie-up with some other products can get them a handful of money.  
Variation = (Standard Deviation/Mean)*100 = (5/20)*100 = 25%
Negatively Skewed
The measure of Central Tendency
95% of the data should lie between μ±2σ
Zero
Median
Continuous Value
Standard deviation is a measure of dispersion
Standard deviation is square root of Variance, so the Standard deviation will be 12
Feedback of 100 customers about your website, rest all are discrete
All of these are measures of central tendency
Deciles divide the complete dataset in a group of 10 parts
Arithmetic mean of the maximum and minimum values of a dataset is called mid-range
25-75th percentile is called IQR i.e. Inter Quartile Range
Machine learning algorithm suits well for small data and it might take huge amount of time to train for large data. Whereas Deep learning algorithm takes less amount of data to train due to the help of GPU(Parallel Processing).
Apply step function, which calculates the AIC for different permutation and combination of features and provides the best features for the dataset.
Feature engineering is the process of using domain knowledge of the data to create features for machine learning algorithm to work -Adding more columns (or) removing columns from the existing column -Outlier Detection -Normalization etc
Python uses most of the Object Oriented programming concepts. But we can also do functional programming in Python. As per the opinion of experts, Python is a multi-paradigm programming language.
We can do functional, procedural, object-oriented, and imperative programming with the help of Python.
In a normal dictionary in Python, there is no order maintained between keys. To solve this problem, we can use OrderDict class in Python. This class is available for use since version 2.7.
It is similar to a dictionary in Python, but it maintains the insertion order of keys in the dictionary collection.
A Module is a script written in Python with import statements, classes, functions, etc. We can use a module in another Python script by importing it or by giving the complete namespace.
With Modules, we can divide the functionality of our application in smaller chunks that can be easily managed.
Python provides // operator to perform floor division of a number by another. The result of // operator is a whole number (without decimal part) quotient that we get by dividing the left number with the right number.
It can also be used floordiv(a,b).
E.g.
10// 4 = 2
-10//4 = -3
In Python, we have a built-in function zip() that can be used to aggregate all the Iterable objects of an Iterator.
We can use it to aggregate Iterable objects from two iterators as well.
E.g.

list_1 = ['a', 'b', 'c']
list_2 = ['1', '2', '3'] for a, b in zip(list_1, list_2):
print a, b

Output:
a1
b2
c3
By using zip() function we can divide our input data from different sources into fixed number of sets.
None is a reserved keyword used in Python for null objects. It is neither a null value nor a null pointer. It is an actual object in Python. But there is only one instance of None in a Python environment.
We can use None as a default argument in a function.

During comparison we have to use “is” operator instead of “==” for None.
A metaclass in Python is also known as class of a class. A class defines the behavior of an instance. A metaclass defines the behavior of a class.

One of the most common metaclass in Python is type. We can subclass type to create our own metaclass.

We can use metaclass as a class-factory to create different types of classes.
The arithmetic mean of the two quartiles is called mid hinge
An upper cap contains the values which fall between 75th percentile and 75th Percentile+1.5*IQR. Similarly, lower cap contains the values which fall between 25th Percentile and 25th Percentile-1.5*IQR
Any value which is more than upper cap and less than the lower cap will fall under the definition of an outlier
Quartile divides a group of the dataset into four subgroups
Random error
18
from sklearn import linear_model import numpy as np
xl = [3,4,5,6,7]
x = np.asarray(xl).reshape(-1, 1)
y = [15,22,25,33,40]
lm = linear_model.LinearRegression()
lm.fit(x, y)
print(lm.intercept_)
print(lm.coef_[0])
Speed of train relative to boy= `(60 + 6)` km/hr = 66 km/hr = [66 xx 5/18]` m/sec = `[55/3]` m/sec.
Time taken to pass the boy=` [120 xx 3/55]` sec = 6.54 seconds
Let be the 4% of a is 4a/100. Since this equals 8, we have 4a/100=8. Solving for a yields a=8×`(100/4)`=200. Also, 8% of b equals 8b/100, and this equals 4.
Hence, we have `(8/100)`×b=4. Solving for b yields  b = 50. Now, c=`b/a`=`50/200`=`1/4.`
`speed =( 6**5/18)m/sec) = (30/18) m/sec`
Time taken by P to cover 100 m = `(100 ** 18/30)`m/sec =60 sec
Time taken by Q to cover 92 m = (60 + 8) = 68 sec.
Q's speed =`("Distance"/"Time"**18/5) kmph =(92/68 ** 18/5)` kmph = 4.86kmph.
Speed = `[142/6]` m/sec = `[23.6 ** 18/5]` km/hr = 84.9 km/hr
Sum of present ages of A, B and C is = 92 years
Therefore , Sum of their ages 4 years ago = 92 – (4 * 3)= 80 years.
4 years ago ratio of the ages of A , B and C was = 1:2:3
Therefore, A’s age four years ago = `1/6` * 80 = 13.3 years.
So , A’s present age =13.3 + 4 = 17.3 years
Gain%"=("Gain"/"C.P")*100=20%
Missing Values
Noise in the Data Set
Outliers
Mixture of Different Languages (like English and Chinese)
Range Constraints
Fraud detection
Disease screening
Imbalanced Data Set means that the population of one class is extremely large than the other (Eg: Fraud – 99% and Non-Fraud – 1%) Imbalanced dataset can be handled by either oversampling, undersampling and penalized Machine Learning Algorithm.
Machine learning algorithm suits well for small data and it might take huge amount of time to train for large data. Whereas Deep learning algorithm takes less amount of data to train due to the help of GPU(Parallel Processing).
Linear Regression and Logistic Regression
Decision Trees and Random Forest
SVM
Naïve Bayes
XGBoost
Apply step function, which calculates the AIC for different permutation and combination of features and provides the best features for the dataset.
Feature engineering is the process of using domain knowledge of the data to create features for the machine learning algorithm to work
-Adding more columns (or) removing columns from the existing column
-Outlier Detection
-Normalization etc
It exists when 2 or more predictors are highly correlated with each other.
Example: In the Data Set if you have grades of 2nd PUC and marks of 2nd PUC, Then both give the same trend to capture, which might internally hamper the speed and time.so we need to check if the multicollinearity exists by using VIF(variance Inflation Factor).
Note: if the Variance Inflation Factor is more than 4, then multicollinearity problem exists.
In Logistic Regression, we can use step() which gives AIC score of a set of features
In Decision Tree, We can use information gain(which internally uses entropy)
In Random Forest, We can use varImpPlot
Measure how much the variance of the estimated regression coefficients are inflated as compared to when the predictor variables are not linearly related.
Parametric machine learning algorithm– Linear Regression, Logistic Regression Non-Parametric machine learning algorithm – Decision Trees, SVM, Neural Network
The algorithm which does not make strong assumptions are a non-parametric algorithm and they are free to learn from training data. The algorithm that makes strong assumptions are parametric and it involves
select the form for the function and learn the coefficients
It works better when we remove the attributes which are unrelated to the output variable and highly co-related variable to each other.
False Positive – A cancer screening test comes back positive, but you don’t have cancer
False Negative – A cancer screening test comes back negative, but you have cancer
True Positive – A Cancer Screening test comes back positive, and you have cancer
True Negative – A Cancer Screening test comes back negative, and you don’t have cancer
We can use the unit testing modules unittest or unittest2 to create and run unit tests for Python code.

We can even do automation of tests with these modules. Some of the main components of unittest are as follows:

Test fixture: We use test fixture to create preparation methods required to run a test. It can even perform post-test cleanup.

Test case: This is main unit test that we run on a piece of code. We can use Testcase base class to create new test cases.

Test suite: We can aggregate our unit test cases in a Test suite.

Test runner: We use test runner to execute unit tests and produce reports of the test run.
A Docstring in Python is a string used for adding comments or summarizing a piece of code in Python.

The main difference between Javadoc and Docstring is that docstring is available during runtime as well. Whereas, Javadoc is removed from the Bytecode and it is not present in .class file.

We can even use Docstring comments at run time as an interactive help manual.

In Python, we have to specify docstring as the first statement of a code object, just after the def or class statement.

The docstring for a code object can be accessed from the '__doc__' attribute of that object.
Sensitivity means “proportion of actual positives that are correctly classified” in other words “True Positive”
Specificity means “proportion of actual negatives that are correctly classified” “True Negative”
If you are dealing with a classification problem like (Yes/No, Fraud/Non-Fraud, Sports/Music/Dance) then use Logistic Regression.
If you are dealing with continuous/discrete values, then go for Linear Regression.
Imputation algorithm means “replacing the Blank values by some values)
Mean imputation
Median Imputation
MICE
miss forest
Amelia
The analogous metric of adjusted R² in logistic regression is AIC.
AIC is the measure of fit which penalizes model for the number of model coefficients. Therefore, we always prefer the model with minimum AIC value.
E(S) = 8/10log(8/10) – 2/10log(2/10)
Note: Log is à base 2
In Machine Learning. Perceptron is an algorithm for supervised classification of the input into one of several possible non-binary outputs
In machine learning, we create models to determine a conclusion. Whenever a model becomes over-complicated to predict a specific set of data, it is called overfishing.

In such a scenario, the initial data is predicted with high accuracy by the model. But with any additional data, the model predicts with much lesser accuracy. This defeats the purpose of the model.

E.g. Let say, we want to predict the type of fruit-based on its height, width, color, and weight. There may be some outliers in our data. Like- a yellow color apple.

If we make our model complicated, it may accurately predict such an object of yellow color to be an apple. But in actual data there maybe lemons. Due to overfitting, our model will start predicting lemons as apples based on color.

One simple way to understand overfitting is that our information from past experiences can be divided into two groups.

1. Information that is relevant for prediction

2. Information that is irrelevant for prediction. It is also called noise.

Whenever there is more noise, it is more difficult for a model to predict correctly. It is a difficult problem for a model to determine which part should be ignored. Once we have a robust learning algorithm, the chance of fitting noise reduces drastically.
Keep the attributes/Columns which are really important
Use K-Fold cross-validation techniques Make use of drop-put in case of neural network
How the root node is predicted in Decision Tree Algorithm?
Mathematical Formula “Entropy” is utilized for predicting the root node of the tree.
Overfitting occurs when the criteria used for training the model are not the same as the criteria used for judging the efficacy of the model.

Overfitting also happens when a model tries to memorize the training data instead of learning from training data.

Once we come to an optimum set of parameters for the model, the overfitting stops. If we increase the number of parameters beyond the optimum level, overfitting occurs.

If our model performs better on training set than on test set, it means there is overfitting in our model. In such a scenario we have high variance in our model. To reduce overfitting, we can find ways to reduce variance in the model. .
TensorFlow
Theano
CNTK
Q126. Name Few Deep Learning
Algorithm
TensorFlow
Theano
Lasagne
mxnet
blocks
Keras
CNTK
TFLearn
Using Stratified Shuffle package
Ensemble Model is a combination of Different Models to predict correctly and with good accuracy.
Ensemble learning is used when you build component classifiers that are more accurate and independent from each other.
SVM can be used if the data is outlier free whereas Naïve Bayes can be used even if it has outliers (since it has a built-in package to take care).
SVM suits best for Text Classification Model and Random Forest suits for Binomial/Multinomial Classification Problem.
Random Forest takes care of overfitting problem with the help of tree pruning
Self Driving Cars
Image Classification
Text Classification
Search Engine
Banking, Healthcare Domain
Logistic Regression
Linear Regression
27
The order of precedence is ** then *.
Thus 3**2 = 9 and then 9*3 = 27.
tup = (‘the’,’Data’,’Monk’)
list_example = list(tup)
print(list_example)
[‘the’,’Data’,’Monk’]
str="TheDataMonk"
print (str)
print (str*2)
print (str[2:5])
print (str[3:])
print (str + ".com")
print ("www."+str+".com")
TheDataMonk
TheDataMonkTheDataMonk
eDa
DataMonk
TheDataMonk.com
www.TheDataMonk.com
Step 1: Put the numbers in order. 1, 2, 5, 6, 7, 9, 12, 15, 18, 19, 27.
Step 2: Find the median. 1, 2, 5, 6, 7, 9, 12, 15, 18, 19, 27.
Step 3: Place parentheses around the numbers above and below the median. Not necessary statistically, but it makes Q1 and Q3 easier to spot. (1, 2, 5, 6, 7), 9, (12, 15, 18, 19, 27).
Step 4: Find Q1 and Q3 Think of Q1 as a median in the lower half of the data and think of Q3 as a median for the upper half of data. (1, 2, 5, 6, 7),  9, ( 12, 15, 18, 19, 27). Q1 = 5 and Q3 = 18.
Step 5: Subtract Q1 from Q3 to find the interquartile range. 18 – 5 = 13.
You survey households in your area to find the average rent they are paying. Find the standard deviation from the following data: $1550, $1700, $900, $850, $1000, $950. Step 1: Find the mean: ($1550 + $1700 + $900 + $850 + $1000 + $950)/6 = $1158.33
Step 2: Subtract the mean from each value. This gives you the differences: $1550 – $1158.33 = $391.67 $1700 – $1158.33 = $541.67 $900 – $1158.33 = -$258.33 $850 – $1158.33 = -$308.33 $1000 – $1158.33 = $158.33 $950 – $1158.33 = $208.33
Step 3: Square the differences you found in Step 3: $391.672 = 153405.3889 $541.672 = 293406.3889 -$258.332 = 66734.3889 -$308.332 = 95067.3889 $158.332 = 25068.3889 $208.332 = 43401.3889
Step 4: Add up all of the squares you found in Step 3 and divide by 5 (which is 6 – 1): (153405.3889 + 293406.3889 + 66734.3889 + 95067.3889 + 25068.3889 + 43401.3889) / 5 = 135416.66668  
Step 5: Find the square root of the number you found in Step 4 (the variance): √135416.66668 = 367.99
The standard deviation is 367.99.
Some of the popular uses of Inductive machine learning are as follows:

Disease Diagnosis: Let say x is the symptoms of a patient.
Then f(x) is the disease patient is suffering from. We can derive the function f() by Inductive machine learning.

Credit risk analysis: Let say x is the important financial indicators. Eg. Social Security, Credit score, etc. of a person.
The function f(x) can give the result whether a credit is approved for this person. In this case, f is based on the machine learning model.

Self Driving Car: In this case images from different cameras of a car are X.
The f(x) is the angle at which the steering wheel can be turned to follow the path.

Face recognition: Images of different people are in a dataset X.
We can create f(x) model to get the name of a person from dataset X.
The different types of tasks in machine learning are as follows:

a. Anomaly detection: In this task, we identify unusual data records also known as outliers. These can be interesting data points or data errors that may need further investigation.

b. Association rule learning: In this task, we search for any kind of relationship between variables in data. It is also known as dependency modeling. E.g. we may find that the price of the house may be impacted by its age.

c. Clustering: In this task, we identify the groups and any kind structure in data. E.g. If we look at the fruits data, we find that the Watermelon is very heavy and it can be easily distinguished by its weight only. Whereas oranges and bananas can be distinguished on the basis of shape.

d. Classification: In this task, we apply the known structure to new data. Once we learn distinguishing characteristics from the clustering task, we can use it to classify the data set. E.g. In fruits example, we can use weight to classify watermelon from the rest of the fruits. Then we can use shape to distinguish bananas from the rest of the fruits.

e. Regression: In regression, we try to estimate a function that can provide a relationship between dependent and independent variables.

f. Summarization: This is the last task in which we provide reports and visualization diagrams to represent our understanding of data.
We can use unsupervised learning for data analysis. Some of the main uses are as follows:

Clustering of data: We can use unsupervised learning to find clusters in data.

Features of data: We can find low-dimensional representations and features of data by using unsupervised learning.

Interesting directions and links in data: We can determine interesting directions as well as links in data by unsupervised learning.

Find relationships: We can use unsupervised learning to find relationships and correlation in data.

Outliers: We can use unsupervised learning to identify outliers or novel observations in data.
Given below is a SWOT analysis for an e-commerce website (Flipkart)
A use case is a list of actions or event steps, typically defining the interactions between a role (known in the Unified Modeling Language as an actor) and a system, to achieve a goal. The actor can be a human or a system.

What is use case?
The flowchart shows the complete flow of the system through symbols and diagrams. It makes it easy for the developer and all concerned people to understand the “flow” of the system. This diagrammatic representation illustrates a solution model to a given problem. A flowchart is a formalized graphic representation of a logic sequence, work or manufacturing process, organization chart, or similar formalized structure.

The purpose of a flow chart is to provide people with a common language or reference point when dealing with a project or process.
what is flowchart
Empathy Map can be drawn in and shape and size but the main parts of an empathy map are:-
  • Thinking
  • Seeing
  • Doing
  • Feeling
Sometimes there are 2 more parts of an empathy map i.e. Pain and Gain.
A laptop is a costly product. I am assuming that people buy a laptop only when they needed. That's why i am going to calculate the potential market of laptops in India. Total population of Bangalore = 18Mn ~ 20Mn
Let's divide the population on the basis of age 0–18 Yr - 30% of 20 Mn = 6 Mn -> We can neglect this age group because generally, they don't need a personal laptop and when needed, they prefer to use others laptop.
19–22 Yr - 10% of 20 Mn = 2Mn -> 0.6*2 Mn -> 1.2 Mn (This is the college-age group. Most of the college students need a laptop. Assumed 60% of them own a laptop)
22–50 Yr = 40% of 20 Mn = 8 Mn. 22-50 age group is the working class of society. I have divided this class into 3 major categories. White-collar employees (25%)
Blue-collar employees (50%)
Small business owners (25%)
Assumed 80% and 30% people in the category of white-collar employees and Small business owners respectively own a laptop or PC. We can neglect blue-collar employees.
80% white collar own a laptop or PC -> 1.6 Mn
Small business owners own laptops or PC -> 0.6 Mn
50–80 Yr = 20% = 4 Mn -> we can ignore this age group
Total laptop + PC users in Bangalore = 1.2+1.6+.6= 2.4 Mn
Corporate offices/Schools/Computer centers generally have desktop. Lets assume 60% are desktops. Laptops = 40% -> 0.9 Mn
Average life of a laptop = 5 year ( in India ) Number of sold per day in Bangalore = 0.9 Mn/ 365*5 ~ 500 laptops
sq = [x**2 for x in range(10)]
print(sq)
import matplotlib.pyplot as plt
a = ['Apple','Mango','Pineapple']
b = [40,60,50]
plt.bar(a,b)
labels = 'Sachin','Dhoni','Kohli','Dravid'
size = [100,25,70,50]
colors = ['pink','blue','red','orange']
explode = (0.1,0,0,0)
plt.pie(size,explode=explode,labels=labels,colors=colors,autopct='%1.1f%%',shadow=True,startangle=140)
plt.axis('equal')
plt.show()
- ARIMA stands for Auto-Regressive Integrated Moving Average.
Autoregression is a time series model that uses observations from previous time steps as input to a regression equation to predict the value at the next time step. It is a very simple idea that can result in accurate forecasts on a range of time series problems
- Basically ARIMAX gives you an added advantage over Linear Regression and AIRMA model. ARIMAX merges both the concepts i.e. it takes the time series forecasting from ARIMA and then uses other variables to check if these variables have any added advantage over the already predicted value of ARIMA.
So, we can safely assume that ARIMAX is the sum of ARIMA and Linear Regression.
- We could have done more feature engineering to include in the Linear Regression. So, there is a scope to better the already 96% accuracy
There were few more questions on the basics of Statistics, you can easily find the answers to these questions on the internet, the reason behind not answering these 4 questions is that Amazon does not allow us to share knowledge which is easily available on the internet:-
In Python we have two options to copy an object. It is similar to cloning an object in Java.

Shallow Copy: To create a shallow copy we call copy.copy(x). In a shallow copy, Python creates a new compound object based on the original object. And it tries to put references from the original object into copy object.

Deep Copy: To create a deep copy, we call copy.deepcopy(x). In a deep copy, Python creates a new object and recursively creates and inserts copies of the objects from original object into copy object. In a deep copy, we may face the issue of recursive loop due to infinite recursion.  
A lambda expression in Python is used for creating an anonymous function.\
Wherever we need a function, we can also use a lambda expression.

We have to use lambda keyword for creating a lambda expression. Syntax of lambda function is as follows:

lambda argumentList: expression

E.g. lambda a,b: a+b

The above-mentioned lambda expression takes two arguments and returns their sum. We can use lambda expression to return a function.

A lambda expression can be used to pass a function as an argument in another function.
In Python, we use range(0,10) to create a list in memory for 10 numbers.

Python provides another function xrange() that is similar to range() but xrange() returns a sequence object instead of list object. In xrange() all the values are not stored simultaneously in memory. It is a lazy loading based function.

But as per Python documentation, the benefit of xrange() over range() is very minimal in regular scenarios.
As of version 3.1, xrange is deprecated.
Python provides many built-in functions that are surrounded by _ symbol at the start and end of the function name. As per Python documentation, double _ symbol is used for reserved names of functions.

These are also known as System-defined names.

Some of the important functions are

Object._new_

Object._init_

Object._del_
We can use Generator to create Iterators in Python. A Generator is written like a regular function. It can make use of yield statement to return data during the function call. In this way, we can write complex logic that works as an Iterator.

A Generator is more compact than an Iterator due to the fact that _iter_() and next() functions are automatically created in a Generator.

Also within a Generator code, local variables and execution state is saved between multiple calls. Therefore, there is no need to add extra variables like self.index, etc to keep track of iteration.

Generator also increases the readability of the code written in Python. It is a very simple implementation of an Iterator.
An Iterable is an object that can be iterated by an Iterator.

In Python, Iterator object provides _iter_() and next() methods.

In Python, an Iterable object has _iter_ function that returns an Iterator object.

When we work on a map or a for loop in Python, we can use next() method to get an Iterable item from the Iterator.
We can use Slicing in Python to get a substring from a String.

The syntax of Slicing is very convenient to use.
E.g. In following example, we are getting a substring out of the name John.

>>> name="John"

>>> name[1:3]

'oh'

In Slicing, we can give two indices in the String to create a Substring. If we do not give the first index, then it defaults to 0.

E.g.

>>> name="John"

>>> name[:2]

'Jo'

If we do not give second index, then it defaults to the size of the String.

>>> name="John"

>>> name[3:]

'n'
The use of Pass statement is to do nothing. It is just a placeholder for a statement that is required for syntax purpose. It does not execute any code or command.

Some of the use cases for pass statement are as follows:

1.Syntax purpose:

>>> while True:

... pass # Wait till user input is received

2.Minimal Class: It can be used for creating minimal classes:

>>> class MyMinimalClass:

... pass

3.Place-holder for TODO work: We can also use it as a placeholder for TODO work on a function or code that needs to be implemented at a later point of time.

>>> def initialization():

... pass # TODO
We can use the following ways to concatenate multiple strings together in Python:

1. use + operator:
E.g.
>>> fname="John"
>>> lname="Ray"
>>> print fname+lname
JohnRay 2.  use join function:
E.g.
>>> ''.join(['John','Ray'])
'JohnRay'
A Namespace in Python is a mapping between a name and an object. It is currently implemented as Python dictionary.
E.g. the set of built-in exception names, the set of built-in names, local names in a function

At different moments in Python, different Namespaces are created. Each Namespace in Python can have a different lifetime.

For the list of built-in names, Namespace is created when Python interpreter starts.

When Python interpreter reads the definition of a module, it creates a global namespace for that module.

When Python interpreter calls a function, it creates a local namespace for that function.
Some of the built-in data types available in Python are as follows:

Numeric types: These are the data types used to represent numbers in Python.
int: It is used for Integers
long: It is used for very large integers of non-limited length.

float: It is used for decimal numbers.

complex: This one is for representing complex numbers

Sequence types: These data types are used to represent a sequence of characters or objects.

str: This is similar to String in Java. It can represent a sequence of characters.

bytes: This is a sequence of integers in the range of 0-255.

byte array: like bytes, but mutable (see below); only available in Python 3.x

list: This is a sequence of objects.

tuple: This is a sequence of immutable objects.

Sets: These are unordered collections.

set: This is a collection of unique objects.

frozen set: This is a collection of unique immutable objects.

Mappings: This is similar to a Map in Java.

dict: This is also called hashmap. It has key-value pair to store information by using hashing.

Main differences between List and Dictionary data types in Python are as follows:

                 1. Syntax: In a List, we store objects in a sequence. In a Dictionary, we store objects in key-value pairs.



                  2. Reference: In List, we access objects by index number. It starts at 0 indexes. In a Dictionary we access objects by key specified at the time of Dictionary creation.



                  3. Ordering: In a List, objects are stored in an ordered sequence. In a Dictionary objects are not stored in an ordered sequence.



                  4.Hashing: In a Dictionary, keys have to be hashable. In a List, there is no need for hashing.

if salary>20000:
  ​print(“Good Salary”)
elif salary<20000
  ​ print(“Average Salary”)
else
  ​print(“Salary is 20000”)
Text Mining/ Text data mining is used for text analysis. It refers to the process of extracting rich information from text.
Time series analysis comprises methods for analyzing time-series data in order to extract meaningful statistics and other characteristics of the data. Time series forecasting is the use of a model to predict future values based on previously observed values.
The purpose of a flow chart is to provide people with a common language or reference point when dealing with a project or process. It gives a pictorial representation of a problem in a structured way(flow).
EmpNo EName Job MGR HireDate Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct-18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT *
FROM emp
WHERE sal > (SELECT sal FROM emp WHERE EName = ‘Aman’);
EmpNo EName Job MGR HireDate Sal Comm DeptNo
1234 Amit Waiter 8382 19-Oct-18 50000 500 50
5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
SELECT *
FROM emp w, emp m
WHERE w.MGR = m.EmpNo and w.HireDate < m.HireDate
Or
SELECT *
FROM emp w, emp m
WHERE w.EmpNo = m.MGR and w.HireDate < m.HireDate
New data points

a) Date and Day of order

b) Time of order

c) Phone number

d) Pin code
With the vanilla GAN, we cannot control and modify the images generated by the generator of the GAN. So, we use conditional GAN. With conditional GAN, we can control and modify the images generated by the generator of the GAN.
The Wasserstein distance is also commonly known as the Earth Movers (EM) distance. It is used as the distance measure in the optimal transport problems where we need to move things from one configuration to another.
In the GAN, we use sigmoid cross-entropy as the loss function. The problem with the sigmoid cross-entropy loss is that once the fake samples are on the correct side of the decision surface, then gradients tend to vanish even though the fake samples are far away from the true distribution.
To avoid this issue we use the least-squares GAN. In the least-squares GAN, although the fake samples generated by the generator are on the correct side of the decision surface, gradients will not vanish until the fake samples match the true distribution.
The discriminator of DCGAN consists of series of convolutional and batch norm layers with leaky ReLU activations.
First, we take the image and feed the image as an input to the discriminator, then the discriminator performs a series of convolution operation and then classifies whether the image is a fake image generated by the generator or the image is a real image from the training data.
The generator of the DCGAN consists of the convolutional transpose and batch norm layers with ReLU activations.
First, we draw a noise from a normal distribution and feed that as an input to the generator. The generator which is composed of convolutional transpose and batch norm layers takes this noise as an input and generates an image similar to the ones present in the training set.
The discriminative model classifies the data points into their respective classes by learning the decision boundary that separates the classes in an optimal way.
The generative models can also classify the data points, however, instead of learning the decision boundary, they learn the characteristics of each of the classes.
We can generate images just based on the text using an interesting type of GAN called StackGAN. The StackGAN works in two stages:
In the first stage, they generate a basic outline, primitive shapes, and create a low-resolution version of the image.
In the second stage, they enhance the picture generated in the first stage by making it more realistic and then convert them into a high-resolution image.
Cycle GAN is majorly used in the premises where it is difficult to obtain the paired training samples. Some of the interesting applications of CycleGAN include photo enhancement, season transfer, converting the real pictures to the artistic pictures, and so on.
We can perceive InfoGAN as an unsupervised version of conditional GAN. In the conditional GAN, we impose a condition on the generator and discriminator to generate the image we want based on the class labels present in the dataset. When we have the unlabelled dataset then we can use InfoGAN for generating the images we want.
In GAN, we minimize the JS divergence between the generator distribution and the real data distribution. But the problem with the JS divergence is that it is not applicable when there is no overlap or when the two distributions do not share the same support.
So, to avoid this issue, we can use the Wasserstein GAN which uses the Wasserstein distance instead of JS divergence.
The role of the generator is to generate new data points that are similar to the ones present in the training set while the role of the discriminator is to classify the given data points as to whether it is a real data point or it is generated by the generator.
In the Cycle GAN, the generator map the images from the source domain to a random permutation of images in the target domain which can match the target distribution. So, to alleviate this, we use a special type of loss called cycle consistent loss.
GANs are widely used in applications which involve image such as image generation, converting grayscale image to colored image, and so on. When dealing with images, we use CNN instead of a feed-forward neural network since CNN is effective at handling images.
Similarly, instead of using vanilla GAN, we can use DCGAN whose generator and discriminator involve the convnets instead of the feed-forward networks. The DCGAN is very effective at tasks related to the images than the vanilla GANs.
The generator network generates new data points similar to the ones present in the training set. In order to generate a new data point, the generator implicitly learns the distribution of the training set, and based on this implicitly learned distribution the generator generates the new data point.
Since the generator network implicitly learns the distribution of the training set, GANs are often called the implicit density model.
The paragraph vector learns the vector representation of the whole paragraph thus capturing the subject of the paragraph.
Some of the popular uses of supervised learning in ML are as follows:

Classification: We can use supervised learning to determine classes in data. We can predict the class of a data point based on the features of data.

Speech recognition: Supervised learning is used in speech recognition to understand human speech by computers. This is also used in translating speech to text as well as in translating text from one language to another language.

Regression: In Regression, we can determine the value of a dependent variable based on the values of independent variables. Eg. Finding the price of a house based on its sq. ft. and other features by using linear regression.

Marketing: We can use supervised learning to find the lifetime value of a customer in customer analytics.
Naive Bayes (NB) algorithm is a classification technique of machine learning. It is based on Bayes’ theorem. We use it for classifying the input data into distinct classes.

As per the NB algorithm, the classifier assumes that all the features used for prediction are independent of each other.

E.g. fruit can be predicted as an orange based on its weight, height, and color. NB algorithm does not try to find any relationship between these features. It uses all the three features independently to determine that this fruit is orange.

NB algorithm is very easy to build. It is very useful for large data sets.

NB is considered one of the most significant and reliable algorithms of classification.
Some of the advantages of Naive Bayes (NB) classifier are as follows:

Simple: NB is very simple to implement. It just needs calculation of the probability of each feature and then it uses all these probabilities to determine the class of a data point.

Large Dataset: NB can be used on very large data sets. Since NB is simple to use, it requires very little computation power compared to other complex algorithms.

Performance: Compared to other complex models, NB provides a very reliable classification.

Training Data: NB requires less training data compared to other algorithms.
We can use Pattern recognition in the following areas:
 
    • Data mining
 
    • Speech recognition
 
    • Statistical analysis
 
    • Computer vision
 
  • Bio-informatics
We use Model selection to determine the most suitable model to solve our problem. In Machine learning, there are many models.

We can create these models to solve a problem. But these models provide different results.

We evaluate the performance of a model based on these results. The criteria for these results can be accuracy, precision, recall, or any other derived score.

In certain cases, precision is more important. E.g. In the case of user experience, we emphasize precision.

In certain cases recall is more important. E.g. In the case of disease detection, it is better to pick a model with less precision but a lower false-negative rate.
We can use the following methods to prevent overfitting in Machine learning.

Cross-validation: We can divide initial data into multiple mini test-train data splits. These mini splits can be used for tuning the model as per requirements. We can use k-fold cross-validation.

Gather More Training Data: We can use more training data to tune the model. If data is not available, we can find ways to gather more data.

Data Augmentation and Noise: If it is not possible to gather more training data, we can augment existing data to make it appear more diverse. It makes the model less prone to overfitting.

Model Simplification: We can simplify the model to use less features. This can lead to a model simple enough to not overfit and make it good enough to learn from data.

Regularization: We can use regularization technique to prevent overfitting. We can tune the values of L1 and L2 weights to make the model more general.
Regularization is a technique in Machine learning to prevent the problem of overfitting in statistical models.

This technique discourages Machine learning to form a complex model to avoid overfitting.

We can use Ridge regression or Lasso regression techniques for tuning the Regularization parameter.

With Regularization the amount of variance in the model is reduced. Whereas bias in the model remains unaffected by Regularization.

In a simple Linear regression equation of y = aX + b, b is bias.
Perceptron is a supervised learning algorithm. We use Perceptron for binary classification problems.

It is a simple linear classifier. It uses a linear function to make a prediction about the class of a data set.

In its simplest form it follows following formula:

y = aX + b

If aX b > 0 then class is 1 else class is 0.

Since Perceptron is a linear classifier, it works well when classes can be separated by a straight line.
We can use the following methods for calibration in Supervised learning:

Platt Calibration: In Platt calibration, we transform the outputs of a classification model into a probability distribution over classes. In binary classification, we distribute data over two classes. But sometimes we need prediction about class as well as the probability of certainty about the prediction. By using Platt calibration we can get the probability estimate. It means we get how sure we are of our classification being correct.

Isotonic Regression: It is also known as monotonic regression. We use Isotonic regression to calibrate the linearity imposed by linear regression. In Isotonic regression, we fit an isotonic curve to means of an experimental result. Isotonic regression is not constrained by any concrete form like a linear function in linear regression.
Support Vector Machine (SVM) algorithm can be used for solving following kinds of classification problems:

Binary classification: We can use SVM to create a binary classifier. In SVM, we can find the hyperplane to divide data into two distinct classes.

Multi-class classification: SVM algorithm works well to divide data into multiple classes. In this case, we have to determine multiple hyperplanes that separate the classes. In certain cases classes are not distinguished by a linear function. We can use kernel trick to transform low dimensional input space to higher-order dimensional space. E.g. we can use circle form to distinguish classes.
Support Vector Machine (SVM) algorithm has following pros and cons:

Pros:


  • Clear: SVM provides a clear separation of data into classes. It works well and provides clear margin of separation between classes.


  • High dimension: SVM is effective in classifying data in high dimensional space.


  • Number of dimension: SVM can effectively classify data that has more number of dimensions than the number of samples.


  • Memory: SVM uses a subset of training points in support vectors (decision function). Therefore it is an efficient algorithm from memory perspective.
 


  • Cons:


  • Low performance: SVM has low performance on a large data set, because it requires more training time to calculate and identify support vectors.


  • Noise: SVM cannot perform well when classes are not distinct. High amount of noise or overlapping classes bring down the accuracy of SVM classification.


  • Probability: We don’t get probability estimates with SVM.
In Machine learning, we can use multiple models and algorithms to get better performance in predictions. This approach of using multiple models is known as ensemble learning.

Ensemble learning is a form of supervised learning. We try multiple algorithms and combine their results to get the level of performance acceptable for our problem.

In general, we get better performance from ensemble learning when diverse models are used. Therefore, certain ensemble methods promote the use of diverse models in learning.
Some of the popular types of Ensemble learning methods are as follows:

Bayes optimal classifier

Bootstrap aggregating (bagging)

Boosting

Bayesian parameter averaging

Bayesian model combination

Bucket of models

Stacking
We use stacking as an ensemble learning method in machine learning. It is also known as stacked generalization.

It is a learning algorithm that combines the results of other algorithms to give a better predictive performance.

First we use different types of learning algorithms on original data for prediction. Then we create a combiner algorithm that provides final prediction by using the prediction results of algorithms used in the first step.

One variation of stacking is known as blending. It was used in a famous Netflix prediction competition.

In general, stacking gives better performance than Bayesian model averaging.
Two main paradigms of ensemble learning are as follows:

Sequential ensemble: In this paradigm, we use multiple models in a sequence. The prediction output of one model is used as an input for the next model.

Parallel ensemble: In this paradigm, multiple models are used in parallel. The prediction results from these models are combined to give the final output of the ensemble model.
In ensemble learning, we combine two or more models to build a more accurate model. Boosting and bagging are two approaches used in ensemble learning.

Bagging: Bagging is also known as Bootstrap aggregating. In bagging, we reduce the variance of the model by generating additional test data. Once the size of the data set is increased, we can tune the model to be more immune to variance.

Boosting: Boosting is a two-step algorithm for ensemble learning. In boosting, we use subsets of a dataset to create an average performance model. Then we tune the model on larger data set to boost the performance of the model.

Data partition: In bagging data partition is random. In boosting, miss-classified data is given higher importance.

Goal: In bagging the goal is to reduce the variance in the model. Boosting aims for increasing the prediction accuracy of the model.

Method: We use a random subspace in bagging. Boosting uses a gradient descent method.

Function: Bagging uses the weighted or average function. Boosting uses a weighted majority vote function.
PCA stands for Principal Components Analysis.

KPCA stands for Kernel-based Principal Component Analysis.

ICA stands for Independent Component Analysis.

These are feature extraction techniques used in machine learning.

The primary purpose of PCA, KPCA, and ICA is dimensionality reduction.
We use an Incremental learning algorithm to improve the performance of a model in machine learning.

In this algorithm, input data is continuously used to further train the model. It is a dynamic technique of machine learning.

Some of the examples of incremental learning algorithms are decision trees, artificial neural networks, and incremental SVM.

The goal of Incremental learning is to adapt the model to new data while maintaining the knowledge learned earlier.

Incremental learning is very popular in data streams of Big data.
we use dimension reduction technique in machine learning to perform feature selection or feature extraction.

with dimension reduction we aim to find the important features that can be used for prediction.

dimension reduction process reduces the number of random variables or features under consideration in a machine learning algorithm.

some of the advantages of dimension reduction (DR) are as follows:


a) DR reduces the need for storage and time to perform a machine learning algorithm. it makes the algorithm more efficient.


b) DR removes the multi-collinearity between features. it can improve the performance of the model.


c)DR dr makes it easy to visualize the model in 2d or 3d views.
Machine Learning is used for programming the computers so that they can automatically learn about data and different situations. Based on learning, computers can take decisions to handle different situations.

Machine learning also helps us in understanding more about different data sets. Sometimes, we use machine learning to create segments of different groups in a dataset.

We can also use it to predict trends based on existing data.

One popular use of Machine learning is to predict user behavior based on different models and past user behavior information.

Some of the examples of Machine Learning use cases are:
  • Email/Spam filtering
  • Network intrusion detection
  • Optical Character Recognition (OCR)
  • Ranking Data
  • Fraudulent Financial Transactions Detection
  • ​Image Recognition
Data mining is the process of discovering patterns in a data set.

We perform data mining by using programming methods and algorithms. We can use it to extract useful information from a large amount of raw data. It helps us in making the data understandable and usable.

Machine Learning is a technique to make the computer learn new things without explicitly programming. It is based on pattern recognition, computational learning theory, and artificial intelligence.

Some of the main uses of Machine Learning are predictive analysis and classification.

The important difference here is that in Data mining, we explicitly look for patterns, whereas in Machine learning our algorithm/model identifies the patterns.
In Inductive machine learning, we start with an input sample (x) and an output sample f(x). Our aim is to estimate the function f(x).

In simple words, we try to learn by example. We generalize the samples and estimate the output.

E.g. Let say we want to classify two types of fruits: apple and watermelon. We can measure the height and weight of these fruits in our test data set and label these as apple or watermelon.

Then we give it to our model. Our model can start approximating that things of higher weight and more height are watermelons and lesser weight things are apples. This is an example of inductive learning.

In deductive learning, we learn by actual experiment. We look at these fruits and conclude that watermelons are heavier than apples. Now we apply this rule to our data to deduce that this is a lighter object, therefore it is an apple.

Generally for Disease diagnosis, we use Inductive machine learning. Based on certain symptoms, a model can predict if the disease is present or not.
The three main stages of building a Hypothesis model in Machine Learning are as follows:

a. Model Building: We use different algorithms to build a model. In this stage, we use training data to build a model.

b. Model testing: In this stage, the model is tested by providing sample test data. We can determine the accuracy of the model in this stage. If the model meets our accuracy criteria on test data, we can use it for actual production purposes.

c. Applying the model: Once the model starts working correctly, we apply the model to real data and start using it. We keep on calculating the accuracy of real data to keep the model updated. In case of any drop in accuracy we go back to the model building stage and tune the model.
Some of the basic learning techniques in Machine Learning are as follows:

a. Supervised Learning: In this learning technique we infer a function from a labeled training data. There are a set of examples with input and output. Based on that we try to come up with a model/function to predict future data. An example of supervised learning is the Decision tree.

b. Unsupervised Learning: In Unsupervised learning, we infer a function from unlabeled data. Two common approaches of unsupervised learning are Self-Organizing Map (SOM) and Adaptive Resonance Theory (ART).

c. Semi-supervised Learning: It is a supervised learning type in which we also make use of unsupervised learning approach. We can use labeled as well as unlabeled data in this technique. It is a general opinion that unlabeled data in conjunction with labeled data can improve the accuracy of model.

d. Reinforcement Learning: In Reinforcement learning the basis is behavioral psychology. Software agents are programmed to take actions to maximize the cumulative reward. This concept is also used in Game Theory, Operations Research, and Control Theory.

e. Transduction Learning: It is a Supervised learning approach in which reasoning is from general cases to specific cases. It is preferred over induction. In transduction, we try to get the answers that we need rather than creating general rules.

f. Learning to Learn: It is also known as Meta Learning. In this approach, the learning technique is flexible enough to keep making modifications in the algorithm to learn new scenarios.
The most common approach to Supervised learning is Inductive learning. We start with a large dataset and split it into a training dataset and test dataset.

We start with data x and output of function f(x). The proposed model is run on the training dataset first. Once the model is mature, we use the test dataset for testing it. When we are at a desired level of accuracy, we can use the model on real data.

Some of the popular algorithms of supervised learning are:

Support Vector Machines
Linear Regression
Logistic Regression
Naive Bayes
Decision Tree
K Nearest Neighbor
A classifier in machine learning is a model that takes discrete or continuous values as input and provides a distinct class or discrete value as an output.

E.g. Let say we have a list of emails. The size and content of these emails vary. When we provide these emails to our classifier model, it can classify these as spam or non-spam classes.

A classifier is a supervised learning approach. Initially, we know the distinct classes of data. Then we build a model to predict the class for a set of input data.

Classifiers are very useful in spam identifications, disease identification, etc.
Tableau development environment works as follows,

-Tableau desktop is used for creating visualizations such as charts, tables, and Maps.

-Users can connect to any data source. If multiple tables are used, then these tables can be joined. Data Sources can be named and shared.

-When the data source is created, depending on the data type, Tableau automatically segregates data into measures and dimensions. Data elements can be transformed.

-These data elements are used to create charts, tables, and Maps by simple drag-and-drop of data elements into the development area.

-One visualization is created per “worksheet”. Multiple worksheets make one “dashboard”.

-If the user needs a “story”, worksheets and dashboards can be used to create a story.

- Worksheet, Dashboards, and Story is created by clicking on the icons at the bottom of the screen How does Tableau development environment works?
or by using the menu optionsHow Tableau development environment works?
When Tableau Desktop is installed, Tableau creates a folder called “My Tableau Repository” in the \Documents folder.
The folder contains all the files required by Tableau

Explain My Tableau Repository folder?

• Logs, the folder contains all the issue logs.
• Workbooks contain all the workbooks – twb and twbx files. Save all your workbooks in this folder.
• Datasources, this folder is used to keep all the data source files such as csv, excel, etc.
• Local Data, when custom geocoding is imported, it gets stored in this folder.
• Map sources, Tableau Map Source (.tms) file is stored in this folder.
• Bookmarks, with .tbm file extension, are stored in this folder.
• Shapes, this folder contains all the shapes provided by Tableau. To add your custom shapes, copy custom shapes in an image format and add to a new folder under this folder.
• Preference.tps, the file is used to add custom color palettes.
Tableau desktop is a development environment and provides functionality to develop interactive dashboards.
Give an Overview of Tableau Desktop

1. Data window. Displays information about the data connection and fields in the data source.
2. Analytics. Contains ready-to-use objects for faster analysis of data.
3. Workbook Name. The workbook consists of data connection, worksheet, dashboard, stories, and images. Worksheet name has an extension of .twb. If it is a packaged workbook then the extension is .twbx.
4. Cards/shelves. Views are created by placing fields on the cards or shelves. Mark cards have different shelves such as color, size, label, detail, tooltip. Fields can be placed on these shelves. Changing the Mark type like Automatic, Shape will change these shelves.
5. Toolbar. The toolbar provides quick access to different functionalities such as undo/redo, adding sheets, sorting, displaying labels, and so on.
6. Pills. Fields or calculations on the rows or columns are called Pills. Click on a pill to access pull-down menu options such as filter, Table calculations, etc. The dimension pill is blue in color and Measure pill is green.
7. View/Canvas. This space displays visualization created by the fields placed on the shelves.
8. Filters. A filter shelf is used to place filters that limit the data.
9. Pages. This shelf displays views on different pages. If a dimension is placed on Pages, it creates separate pages for each dimension. If a measure is used, then measure is converted to discreet measure.
10. Show Me. Depending on the field selection in the data window, Tableau suggests the best-suited visualization. Different visualizations can be selected in the "Show Me” box.
11. Status bar. Displays various attributes of the visualization in the current worksheet. It displays information such as the number of Marks, the the number of rows and columns, and aggregated measures.
12. Sheet Name. Displays the name of the current worksheet. Give meaningful names to the sheets, if multiple sheets are created. There are three types of sheets - worksheet, dashboards, and story.
Exercise: Launch Tableau Desktop, connect to data source Sample – Superstore.xls under \Documents\My Tableau Repository\Datasources\. Use Orders datasheet as the source data. Refer to downloaded workbook – Tableau_QA_SolutionsWorkBook.twbx.
Every organization has certain measures that are used to evaluate performance. These measures or indicators are called Key Performance Indicators (KPI). KPI’s are different for different organizations, for example, for a Staffing company, one for the KPI may be # of Consultants working at the client site, for a hospital it may be the number for patients treated.
Tableau can be integrated into source control applications like Microsoft TFS (Team Foundation Server).
Tableau can be used with a variety of data sources. This includes Excel, CSV, Multi-dimensional cubes, MS –access, and also relational databases, cloud-based data sources.

What data sources can be used with Tableau?
Typical Tableau implementation follows these principles,

- Requirement gathering. Discussion with users to understand their data and visualization requirements.
- Analyze data sources. Gather information on different data sources and relationships among the data elements.
- Create mockups of the dashboards. User review of mockups.
- Create a data source.
- Structure the data by renaming data fields to user-friendly columns. As required, create calculated fields, hierarchies, parameters, and other elements.
- Depending on the number of data elements in the data source, organize data into different folders.
- worksheets and dashboards as per the requirement.
- Unit Testing. Comparing the data in the visualizations with the source data. Review of a dashboard by the users.
- If the server option is available and desired deploy dashboards over the Server.
- Create data load schedules.
Dimension tables also referred to as Dimensions contain the descriptive attribute of the data element. Example of a Dimension table will be a Customer Dimension, containing fields that describe a customer, i.e. Customer Name, Address, and so on. Dimensions are used to slice the data. The primary key in a Dimension table also referred to as Dim ID uniquely identifies each row.

The fact table contains a measurable attribute of the data. For example, the Fact table can contain Sales Amount or the Sales made to a Customer. The fact table contains the foreign key of the Dimension Table.

Dimension provides context to a Fact. Without Dimension Fact will not be meaning full.
In the above example, Customer Info will be stored in the Customer Dimension, and Sales made to different customers will be stored in the Fact table. These two tables will be joined to get the Sales Amount of a Customer.

What are dimension and fact Tables?
1) Cardinality between Dimension and Fact is one too many, with many on the Fact side. Look for One to Many relationships in the ER diagram.
2) The dimension contains the textual/ descriptive attribute and Fact contains the measurable data. Look for such data elements in the tables.
3) The fact table contains the foreign key of the dimension table.
Star schema is a data structure in which a Fact table is at the center surrounded by dimension tables. The structure looks like a Star and hence the name. Using below Star schema, you can slice Sales Amount and Quantity by Customer, Time period, Product, and Regions.

What is star schema?
In Star schema, the Fact is at the center and surrounded by dimension tables. A snowflake schema is similar to star schema. In Snowflake schema, dimension table/s may be connected to other related dimensions. For example, the Product dimension may connect to the Category dimension. What is a Snowflake scheme?
When creating a dashboard, it is important to get the requirements from the users. When gathering requirements,

  • -Take data requirements. Data sources, key filters, data refresh frequency.

  • -Ask users about the KPI (Key Performance Indicators).

  • -Inquire about the questions users are trying to answer.

  • -Document business rules required for the dashboards.

  • -Understand security requirements.

  • -Gather details about any existing reports or desired visualization.

  • -Provide layout of the screen in terms of a dashboard- charts, and tables in excel or hand-drawn. Get the user's feedback on the layout.

  • -Get requirements about the screen resolution.
Feature projection (FP) is an approach to dimension reduction in machine learning.

We use FP to transform the data from high dimension space to space with fewer dimensions. Some of the techniques of FP are as follows:

Principal component analysis (PCA): PCA performs a linear mapping of data from higher to lower dimensions.

Non negative matrix factorization (NMF): NMF decomposes a non-negative matrix to product of two non-negative ones.

Kernel PCA: In this technique PCA can be done by using kernel trick. It is used for constructing non-linear mappings.

Autoencoder: We use autoencoder to learn non-linear dimension reduction functions. We use inverse functions in this technique to create original representation from coding.
Support vector machine (SVM) is a supervised learning model of machine learning.

SVM is used for classification and regression problems. It is mostly used in classification analysis to divide the dataset into multiple classes.

In SVM we try to identify the hyperplane by which data can be divided into classes. Then we try to maximize the distance between classes so that data can distinctly be labeled. This distance is called the margin.

Most of the time we get a linear classification in SVM. But sometimes we have to deal with non-linear classification. In such a scenario we can use kernel trick. The kernel trick takes low dimensional input space and takes it to higher-dimensional space.
Bias in the machine learning model comes from simple assumptions about the model. This can lead to under fitting of data. It reduces the accuracy of the model.

Variance in machine learning comes from high complexity in an algorithm. Due to variance, the model becomes sensitive to variation in data. Variance causes the inclusion of noise in the model. This leads to over fitting of data.

Therefore in machine learning, we have to balance the bias and variance so that model provides prediction with optimum accuracy. We do not want a high bias or high variance in our model.

It is an art to maintain a balance between bias and variance while creating a machine-learning model.
KNN is also known as the K nearest neighbor algorithm. KNN is a classification algorithm based on a supervised learning approach. K-means clustering is a clustering algorithm based on an unsupervised learning approach.
We need labeled data in the KNN algorithm. In K-means clustering we need unlabeled points and a threshold.  The algorithm takes unlabeled points and clusters them into groups by calculating the mean distance between the points.
Precision is also known as the positive predictive value in machine learning.

Let say we have a dataset of 12 fruits, apples, and oranges. Our model identifies 8 apples out of these. Out of these 5 apples are actually apples. This is called true positives. These rest 3 apples identified by the model are called false positives. In this case, the precision of the model is 5/8.

Precision tells us how effective is our model in identifying the true positives.

The other way of looking at precision is how useful the prediction is.

Precision is a measure of quality or exactness of results.
Recall in machine learning is also known as sensitivity.

Let say we have a dataset of 12 fruits, apples, and oranges. Our model identifies 8 apples out of these. Out of these 5 apples are actually apples. This is called true positives. These rest 3 apples identified by the model are called false positives. In this case, the recall of the model is 5/12.

The other way of looking at recall is how complete the results are.

Recall is a measure of quantity or completeness of results.

Recall can also be used to find the probability that a relevant result is predicted by model.
We use Bayes’ theorem in the Naive Bayes method of classification. It is a popular model for classifying data.

Bayes’ theorem is a mathematical theorem about probability. We can calculate the probability of an event if we know the probability of events prior to it.

The formula of Bayes’ theorem is:
P(A/B) = ( P(B/A) * P(A) ) / P(B)

Here A and B are events and P(B) is not equal to zero.

Due to the simple use of Bayes’ theorem, the Naive Bayes classifier is simple and highly scalable. It just requires parameters in a linear manner. Some of the variations of

Naive Bayes theorem are:
Gaussian Naive Bayes, Multinomial Naive Bayes, and Bernoulli Naive Bayes.
We use Regularization in machine learning to minimize the overfitting of the model. Two popular techniques of regularization are L1 and L2.

L1 regression is known as Lasso regression. L2 regression is known as Ridge regression.

Both of these techniques impose a penalty on coefficients to minimize overfitting.

L1 regularization is based on a sparse approach. L1 shrinks some coefficients to 0 value. This minimizes the impact of such features in the model. L2 regularization tries to spread the error among all the terms.

If we have data with correlated features, L2 regression works better. If only some features in data are related to prediction then L1 regression gives better results.

Type I and Type II errors are used in machine learning to find the effectiveness of the hypothesis. These are the concepts derived from statistics.

      Type I Error: Type I error is the rejection of the null hypothesis. It is also known as a false positive. It means the result indicates that a condition is present but it is not present. E.g. If a test predicts that a person has diabetes, but in reality, the person does not have diabetes. It is an example of a Type I error.

    Type II Error: Type II error is the failure to reject the null hypothesis. It is also known as a false negative. It means, the result indicates that a condition is not present, but it is actually present. E.g. If a test predicts that a person does not have diabetes, but in reality the person has diabetes. It is an example of a Type II error.

In machine learning, we have to establish the acceptance criteria of a model on the basis of acceptable false-positive and false-negative results. Therefore, Type I and Type II errors are quite useful in machine learning models.

Deep learning is a branch of machine learning. It is related to neural networks. It uses the principles of neuroscience to create the model.

It is also known as Deep structured learning or hierarchical learning.

The main inspiration behind deep learning is information processing and communication patterns in the biological nervous system.

Artificial Neural networks and Deep neural networks are two popular techniques of deep learning.

Some of the most popular uses of Deep learning are speech recognition, image recognition, natural language processing, etc.
Your logical-thinking skills are being put to the test with this question. As you answer, highlight how you thoughtfully respond to changing situations.

One potential response is something along the lines of, “First, I prioritize the changes to requirements, the scope of changes and the impact analysis to the project. Next, I perform an impact analysis of the project cost, timeline, and resources. Finally, I evaluate whether the scope change is introducing new gaps to the technical or functional designs or development and testing.”
The steps generally followed are:-

1)Market Analysis
2)Competitor Analysis
3)SWOT Analysis
4)Strategic Vision and Feature Set
5) Prioritize Features
6) Use Cases
7)Storyboards
8)Test Cases
10) Monitoring Scalability.

These topics are dealt in detail further. This is not the only series of steps, but now you do have a prototype to keep in mind and you can build your own content.
When defining your product strategy be sure to answer the following questions.


  • Who are you selling to?


  • Define your target customer or market. Identify whom you are selling to, and what that market looks like.


  • What are you selling?


  • how potential customers will perceive your product compared to competitive products. Understand what makes your product unique in the market.


  • What value do you provide your customers?


  • Determine what problems your product solves for customers. You cannot be everything to everyone within a particular market, but you can help to solve specific problems. Create a value proposition to position the value you provide and the benefits that customers will receive with your solution.


  • How will you price your product?


  • State how you will price the product. Include its perceived value and a pricing model.


  • How will you distribute your product?


  • how you will sell your product, and how your target market will acquire your product.
    • Take an example of Customer Segment to understand the different parts of an empathy map.

    Thinking and feeling – This section deals with getting in the shoes of the customer and to know what they really want, think and feel for the product. What are their hopes, dreams, and fears?

    Hearing – It maps all the things which the customer hears from different sources e.g. Social Media, Print Media, Customer feedback, etc. Basically what influences the customer!

    Seeing – It looks after what the customer looks at? Everything including her environment, friends, and what the market offers. Pains – The problems felt by the customer in order to purchase a product or receive a service from the company. It could be anything between the problem in logging in to the website to bad service from the company’s side.

    Gain– Now the main part i.e. the customer’s expectation from the company and the product.

    empathy map

    So this map helps you get all the required information about the customer and his views.
    The information gathered from Data Mining and Analysis can be used in the following application:-

    • Fraud Analytics
    • Market Analysis
    • Customer Retention
    • Product Control
    The following are the requirements list:-

    1. Scalability – The method should be able to scale in order to handle a large set of data.
    2. Able to handle different data types
    3. Able to deal with outliers and noise
    4. Usability
    5. Insensitive to the order of input records
    Regression is used for the following:-
    -It is used to derive a significant relationship between variables
    -Regression analysis also allows us to compare the effects of variables measured on different scales, such as the effect of price changes and the number of promotional activities.
    The findings from the above data points could be the following:

    a)You have the pin code so you can try to find the pattern and see if there are some food items that are delivered in a particular area. For eg. If there is a Bengali colony and fish items are being sold a lot, so you can recommend food items

    b) Day can get you the food items which are more popular during weekends

    c) Time of order can get you the most popular food of the hour, you can give some discount on these items or make in bulk d) Phone number to inform about new offers(Not a finding)
    Whether a series displays AR or MA behavior often depends on the extent to which it has been differences.

    An “under differenced” series has an AR signature (positive autocorrelation) After one or more orders of differencing, the autocorrelation will become more negative and an MA signature will emerge 

    Don’t go too far: if series already has zero or negative autocorrelation at lag 1, don’t difference again.


    1-Shelf life of model (For how long is the product good to use)

    2-Nature of data

    3. Nature of business

    4-expected accuracy
    Can you give an example of use case ? or can you define the use case of car rental system ?
    Before drawing an activity diagram we should identify the following elements:

    a) Activities

    b) Association

    c) Conditions

    d) Constraints
    The following is the activity diagram for login

    Give example of an activity diagram or draw an activity diagram?
    Net Present Value (NPV) is the difference between the present value of cash inflows and the present value of cash outflows. NPV is used in capital budgeting to analyze the profitability of a projected investment or project.
    Step 1: Define your strategic focus areas.
    Step 2: Create at least one measurable outcome for each focus area.

    Within each focus area, what does success look like?

    Design Pathfinder Projects with milestones.

    To reach your outcome you will need at least one project. We call these "Pathfinder Projects" because, in truth, you don't know what will work. Just like outcomes, though, you need to be crystal clear in defining a Pathfinder Project.

    The best way to get clarity is by setting commonsense milestones. That way, you can tell if your project is getting off course, or worse yet, not working at all. Your milestones alert you to failure. When you fail early, you fail cheaply...and you learn.

    Draft a short Strategic Action Plan.

    Commit to a 30-day review process


    1. Problem definition
    2. Data Exploration
    3. Data Preparation –
    4. Modeling
    5. Validation
    6. Implementation and tracking
    The first step is to identify variables with missing values. Assess the extent of missing values. Is there a pattern in missing values? If yes, try and identify the pattern. It may lead to interesting insights.

    If no pattern, then we can either ignore missing values (SAS will not use any observation with missing data) or impute the missing values. Simple imputation – substituted with mean or median values.
    You can use different methods to assess how good a logistic model is.

    a. Concordance – This tells you about the ability of the model to discriminate between the event happening and not happening.
    b. Lift – It helps you assess how much better the model is compared to random selection.
    c. Classification matrix – helps you look at the false positives and true negatives.

    Some other general questions you will most likely be asked:
    - What have you done to improve your data analytics knowledge in the past year?
    - What are your career goals?
    - Why do you want a career in data analytics?

    The answers to these questions will have to be unique to the person answering it. The key is to show confidence and give well thought out answers that demonstrate you are knowledgeable about the industry and have the conviction to work hard and excel as a data analyst.
    Case studies usually require you to give a detailed analysis of the situation and to recommend possible future action or a new design. Your marker is looking for how well you analyze the situation and provide a realistic solution to the issues/problems.
    Your marker would also look at the quality and reasonableness of your recommendations. Reasonableness includes that the recommendations could be achieved taking account of resources such as time and money. There are often many solutions to a problem but giving all possible solutions loses marks in a case study situation because it is considered unrealistic.
    Your course description of the task should contain guidelines on the types of questions to ask when analyzing the case along with other issues/possibilities to consider in your analysis and discussion.
    The golden rule to measure the quality of a good requirement is the ‘SMART’ rule. According to this rule, a requirement should be:

    Specific: The requirement should be specific so that it could be properly documented

    Measurable: We should be able to measure the success criteria of the requirement by different parameters

    Attainable: The requirement should be possible to attain with the given resources

    Relevant: The requirement should be in line with the project’s business case

    Timely: The requirement should be posed in time i.e. early in the project life cycle.
    The following are the best practices that are followed to write a clear and well-documented use case:

    1) Capture both functional and non-functional requirements in a use case.
    2) Include use case diagrams along with the use case.
    3) Include the UI details/notes in the use case.
    Since business analysis is an evolving and multifaceted profession, hiring managers want to know that you are aware of the necessary skills for success. You probably have your own list, but make sure to highlight both technical and nontechnical attributes you can bring to the job.
    The job description should provide clues as to what types of skills the employer is looking for on both fronts — especially technical requirements. Learning what you can about the company culture prior to the interview can also provide insight into interpersonal abilities that will likely be valued.
    The 4 main forecasting techniques are:-

    1) Delphi Technique – This technique involves questioning, compilation, and re-questioning until the company reaches a conclusion. The panel of experts are made to prepare some question and they are asked to answer these questions, the answers are then evaluated. After this, the questions are re asked to the panel. This process is carried on in an infinite loop until and unless a conclusion is reached.  

    2) Subjective approach – The subjective approach uses a lot of brainstorming approaches by the core members along with employees to generate ideas and solve problems of the company. This prediction is done based on subjective thoughts and feelings  

    3) Time Series Forecasting – It is a quantitative technique and it involves looking for trends in the business by crunching data on hourly, daily, weekly, monthly, yearly, or longer periods. The increase in cold drink sales in the festive season in India is an example of a seasonal event.  

    4) Scenario Writing - In Scenario Writing, the forecaster generates different outcomes based on different starting criteria. The decision-maker then decides on the most likely outcome from the numerous scenarios presented. Scenario writing typically yields the best, worst, and middle option.
    There are many tests, few are:-


    • A/B Test

    • Student’sT Test

    • Chi-square Test

    • Fisher’s Exact Test

    • Mann-Whitney Test
    The following are the steps for KDD:-

    1. Data Cleaning – Clean the data and remove noise and inconsistency
    2. Data Integration – Combine multiple data source
    3. Data Selection – Relevant data is retrieved from the data set
    4. Data Transformation – Data is transformed in a useable format to solve the problem statement
    5. Data Mining – Intelligent methods are applied in order to extract data pattern
    6. Pattern Evaluation – Data patterns are evaluated
    7. Knowledge Presentation – Showcase the evaluated pattern
    The following are the different data clustering algorithms:-

    1. Distribution Based Clustering
    2. Density-Based Clustering
    3. Connectivity Based Clustering
    4. Centroid Based Clustering (K-Mean)
    A flowchart is a type of diagram which represents an algorithm or flow or process showing the steps as boxes of various kinds, and their order by connecting them with arrows. Flowcharts are used in designing and documenting simple processes or programs. The two most common types of boxes in a flowchart are:

    -a processing step, usually called activity and denoted as a rectangular box

    -a decision usually denoted as a diamond.
    INVEST assists a project manager and technical team to get a quality product/services/scrum meeting. INVEST stands for

    I – Independent ( The product should be independent/minimum dependent on other products)

    N – Negotiable(The product should not be fixed, it can be rewritten or even discarded, depending on the business, market, technical, or any other type of requirement by team members.)

    V – Valuable(bring actual project-related value to the end-user and continuously deliver a quality product to the users)

    E – Estimable (Product need to be estimable in order to plan its flow, iterate )

    S – Sized Appropriately (The size of the product should suit the environment of production i.e. number of resources, the technology required, etc.)

    T – Testable (The product/service should be testable in order to reiterate ).
    A case study is a "published report about a person, group, or situation that has been studied over time "If the case study is about a group, it describes the behavior of the group as a whole, not the behavior of each individual in the group.

    Case studies are analyses of persons, events, decisions, periods, projects, policies, institutions, or other systems that are studied holistically by one or more methods. The case that is the subject of the inquiry will be an instance of a class of phenomena that provides an analytical frame — an object — within which the study is conducted and which the case illuminates and explicates

    -> Case study/Analysis is the basic requirement to crack any BA interview

    Red chilly restaurant is a Giant food chain in Bangalore with delivery and dinning. They are earning good but they want more ( greedy !!). They have heard a lot about data scientists, so they hired one. The restaurant had been collecting data on its service for the last 10 years. The data they had was too little but clean. We managed to get a glance on the data set

    -Name

    -Sex

    -Food item code

    -Pin code

    You being the new recruiter have to answer a few questions and make a recommendation engine to recommend 2 food items to a person new to the restaurant.
    Data exploration is done to become familiar with the data. This step is especially important when dealing with new data. There are a number of things you will want to do in this step –

    a.        What is there in the data – look at the list of all the variables in the data set. Understand the meaning of each variable using the data dictionary. Go back to the business for more information in case of any confusion.

    b.        How much data is there – look at the volume of the data (how many records), look at the time frame of the data (last 3 months, last 6 months, etc.)

    c.        Quality of the data – how much missing information, quality of data in each variable. Are all fields usable? If a field has data for only 10% of the observations, then maybe that field is not usable, etc.

    d.        You will also identify some important variables and may do a deeper investigation of these. Like looking at averages, min and max values, maybe 10th and 90th percentile as well.

    e.        You may also identify fields that you need to transform into the data prep stage.
    In data preparation, you will prepare the data for the next stage i.e. the modeling stage. What you do here is influenced by the choice of technique you use in the next stage.

    But some things are done in most cases – for example, identifying missing values and treating them, identifying outlier values (unusual values) and treating them, transforming variables, creating binary variables if required, etc.

    This is the stage where you will partition the data as well i.e. create training data (to do modeling) and validation (to do validation).
    These types of questions are always on the list. Do prepare for such questions as descriptive as possible. The sample answer could be “I have a good combination of strong technical skills, and the ability to build strong customer relationships. This allows me to use my knowledge and breakdown information to be more user-friendly”
    A business analyst must possess fundamental skills such as elicitation skills, problem-solving skills, communication, and management skills. Alongside this, he must have knowledge of IT skills, Software development understanding, and domain knowledge regarding the domain he is working in.
    A/B testing is a form of statistical hypothesis testing with two variants leading to the technical term, two-sample hypothesis testing, used in the field of statistics. In simple words, A/B Testing in web analytics is used to compare the performance of 2 web design to get a better design. Suppose you have 2 designs to display an advertisement on your website, one being a picture and another a text or link. So, you can compare the performance of the two designs by A/B Testing.
    A recommendation engine is a system that predicts the preference of a user depending on various factors like geography, previously bought an item, cookie, customer clustering, etc. An algorithm that analyzes a customer’s purchases and actions on an e-commerce site and then uses that data to recommend complementary products.
    A test applied to data for atomicity, consistency, isolation, and durability
    ARIMA stands for Autoregressive Integrated Moving Average. It is a way of forecasting data based on historical data. There are three attributes of this method:-

    a) Autoregression (p) Order of the autoregressive model

    b)Integration(d) degree of differencing

    c)Moving Average(q) order of Moving Average Model
    Activity diagram is basically a flow chart to represent the flow from one activity to another activity. The activity can be described as an operation of the system. So the control flow is drawn from one operation to another. This flow can be sequential, branched, or concurrent. Activity diagrams deal with all types of flow control by using different elements like a fork, join, etc.
    The purpose of an activity diagram is to provide an outline of workflow in the business, including the action and activities that are completed. For example, with a company, there is likely to be more than one department, with various access levels to the system. If there are departments including HR, Medical, and Accounting, they only have access to the screens that relate to their work. An activity diagram will be used to highlight the differences in the departments, which is extremely helpful for developers when they are coding and designing.
    Competitor analysis in marketing and strategic management is an assessment of the strengths and weaknesses of current and potential competitors. This analysis provides both an offensive and defensive strategic context to identify opportunities and threats.
    Big Data as the name suggests is a large chunk of data with millions of rows and multiple columns. These are different from normal data because the data is so large that traditional data processing applications are inadequate.

    Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. But it's not the amount of data that's important. It's what organizations do with the data that matters.
    Clustering is the process of grouping objects in such a way that objects in one group is much similar to objects of that group than to those in other groups. We first partition the set of data into groups based on data similarity and then assign the labels to the groups.
    Business analysis is the set of tasks and techniques used to work as communication among stakeholders in order to understand the structure, policies, and operations of an organization, and to recommend solutions that enable the organization to achieve its goals.
    D.B.M.S. stands for Database Management System which is a computer software application that interacts with the customers and other applications.

    A database management system (DBMS) is a computer program (or more typically, a suite of them) designed to manage a database, a large set of structured data, and run operations on the data requested by numerous users.

    You can name a few DBMS if you have used it before. I had used MySQL, Microsoft Access, Teradata (For Big data), etc.
    The act of reviewing and revising data to remove duplicate entries, correct misspellings, add missing data, and provide more consistency.
    Data integration is the combination of technical and business processes used to combine data from disparate sources into meaningful and valuable information. A complete data integration solution delivers trusted data from a variety of sources.
    Data mining is the practice of automatically searching large stores of data to discover patterns and trends that go beyond simple analysis. Data mining uses sophisticated mathematical algorithms to segment the data and evaluate the probability of future events. Data mining is also known as Knowledge Discovery in Data (KDD).
    Data management incorporates the following practices needed to manage the full data lifecycle in an enterprise:


    • -data governance

    • -data architecture, analysis, and design

    • -database management

    • -data security management

    • -data quality management reference and master

    • -data management

    • -data warehousing and business intelligence management

    • -document, record, and content management

    • -metadata management

    • -contact data management
    A data model defines the structure of the data for the purpose of communicating between functional and technical people to show data needed for business processes, or for communicating a plan to develop how data is stored and accessed among application development team members.
    Data is simply facts, figures, numbers, etc. It is the raw material that is processed to get some meaningful information. When data are processed, interpreted, organized, structured, or presented so as to make them meaningful or useful, they are called information.
    E.g. the Test Score of each student is data. The average score of the whole class is information.
    Data Warehouse is constructed by integrating data from multiple heterogeneous sources. It supports analytical reporting, structured and/or ad hoc queries, and decision making.
    The database is a collection of information that is organized so that it can be easily accessed, managed, and updated. In one view, databases can be classified according to types of content: bibliographic, full-text, numeric, and images.
    An empathy map is basically used by companies to get a deeper insight into their customers. Much like a user persona, an empathy map can represent a group of users, such as a customer segment. Empathy Map can represent a group of customers.
    A decision tree is a graph to represent choices and their results in the form of a tree. The nodes in the graph represent an event or choice and the edges of the graph represent the decision rules or conditions. It is mostly used in Machine Learning and Data Mining applications using R.
    Forecasting is the process of predicting the future of a product or company based on the performance in the past and present data.
    It describes how a business analyst identifies a business need, refine and clarify the definition of that need, and define a solution scope that can feasibly be implemented by the business.
    K-mean is a simple algorithm that is used for data clustering. The procedure follows a simple and easy way to classify a given data set through a certain number of clusters (assume k clusters) fixed a priori.

    The main idea is to define k centroids, one for each cluster. These centroids should be placed in a cunning way because of different location causes the different result. So, the better choice is to place them as much as possible far away from each other. The next step is to take each point belonging to a given data set and associate it to the nearest centroid.

    When no point is pending, the first step is completed and an early grouping is done. At this point, we need to re-calculate k new centroids as the barycenter of the clusters resulting from the previous step. After we have these k new centroids, a new binding has to be done between the same data set points and the nearest new centroid. A loop has been generated.

    As a result of this loop, we may notice that the k centroids change their location step by step until no more changes are done. In other words, centroids do not move anymore.
    The Internet of Thing is a system of interrelated computing devices, mechanical and digital machines, objects, animals, or people that are provided with unique identifiers and the ability to transfer data over a network without requiring human-to-human or human-to-computer interaction.
    Basically there are two types of variables in a regression equation. The one whose value is found from experiments is called the predictor variable. The one whose value is to be calculated is the response variable. In linear regression, the strength of relationship between response variable and predictor variable is calculated.
    Y=ax+b
    Y is response variable
    X= Predictor variable
    A and b are constants
    Sometime a column can have an only Boolean variable in it(True/False or 0/1) etc. and you need to know the relationship between the response variable(Boolean value) with predictor variable. The Logistic Regression is a regression model in which the response variable (dependent variable) has categorical values such as True/False or 0/1. It actually measures the probability of a binary response as the value of the response variable based on the mathematical equation relating it with the predictor variables.

      y = 1/(1+e^-(a+b1x1+b2x2+b3x3+...)
    A marketing analysis is a study of the dynamism of the market. It is the attractiveness of a special market in a specific industry. Marketing analysis is basically a business plan that presents information regarding the market in which you are operating in. It deals with various factors.
    Multiple regression is an extension of linear regression into the relationship between more than two variables. In simple linear relation, we have one predictor and one response variable, but in multiple regression, we have more than one predictor variable and one response variable.

    Y=a1x1+a2x2+..+b
    A modeling language is any artificial language that can be used to express information or knowledge or systems in a structure that is defined by a consistent set of rules. The rules are used for interpretation of the meaning of components in the structure.

    -Behavior Trees are a formal, graphical modeling language used primarily in systems and software engineering.

    -A flowchart is a schematic representation of an algorithm or a stepwise process.
    The process of analyzing multidimensional data using three operations: consolidation (the aggregation of available), drill-down (the ability for users to see the underlying details), and slice and dice (the ability for users to select subsets and view them from different perspectives).
    Online Analytical Processing (OLAP) is a technology that is used to organize large business databases and support business intelligence. It performs a multidimensional analysis of business data and provides the capability for complex calculations, trend analysis, and sophisticated data modeling.
    The product strategy forms the basis for executing a product roadmap and subsequent product releases. The product strategy enables the company to focus on a specific target market and feature set, instead of trying to be everything to everyone.
    It is a decision-making technique, also known as 80/20 rule.  It is used for quality control and defect resolution. It explains a few factors that can be responsible for big problems. It is named as 80/20 rule, because as per this rule, 80 % of effects in the system, arises from 20 % causes.
    In the random forest approach, a large number of decision trees are created. Every observation is fed into every decision tree. The most common outcome for each observation is used as the final output. A new observation is fed into all the trees and taking a majority vote for each classification model.

    An error estimate is made for the cases which were not used while building the tree. That is called an OOB (Out-of-bag) error estimate which is mentioned as a percentage.
    Regression is a form of predictive modeling technique to determine the strength of the relationship between a dependent and independent variable. One of these variables is called a predictor variable whose value is gathered through experiments. The other variable is called the response variable whose value is derived from the predictor variable.

    Y=aX+b

    Linear regression (X is predictor variable and Y is response variable)
    Revenue can most easily be thought of as the top line of an income statement or profit and loss statement. Revenue is the total amount of income generated by the sale of goods or services related to the company's primary operations. If the company is a shoe retailer, the money it makes from selling shoes before accounting for any expenses is its revenue.
    Requirement analysis describes how business analysts prioritize and progressively elaborate stakeholder and solution requirements in order to enable the project team to implement a solution that will meet the needs of the sponsoring organization and stakeholders.
    SWOT stands for Strength, Weakness, Opportunity, and Threat. It is a structured planning method that evaluates those four elements of a project or business venture. It can be carried out for a huge range from a company, product, place, industry to person.

    1)Strengths: characteristics of the business or project that give it an advantage over others
    2)Weaknesses: characteristics that place the business or project at a disadvantage relative to others
    3)Opportunities: elements that the business or project could exploit to its advantage
    4)Threats: elements in the environment that could cause trouble for the business or project
    Student’s T Test is used to determine if the two sets of data are significantly different from each other.

    Example - Comparison of analytical results obtained with the same method on samples A and B, in order to confirm whether both samples contain the same percentage of the measured analyte or not.

    The outcome of these tests is the acceptance or rejection of the null hypothesis (H0). The null hypothesis generally states that: "Any differences, discrepancies, or suspiciously outlying results are purely due to random and not systematic errors". The alternative hypothesis (Ha) states exactly the opposite.
    The application of statistical functions on comments people make on the web and through social networks to determine how they feel about a product or company
    Application software that is used over the web by a thin client or web browser. Salesforce is a well-known example of SaaS.
    Data Mining process includes the following 4 stages:-

    1.       Problem Definition - This step is used to clear the definition for which data mining is to be done. Defining the problem is the most important part of the complete process. The whole data extraction and analysis depends on the data required. If you need to know which store is performing poorly in a locality then you need data on the store level.

    2.       Data Collection - The data is extracted from a humongous amount of data. The correct data need to be extracted from the raw data. It is then cleaned and sampled as well. Cleaning the data protects the analysis from exception handling.

    3.       Model Building - In this phase, you select and apply various modeling techniques and calibrate the parameters to optimal values. It is time to evaluate how well the model satisfies the originally-stated business goal (Stage 1)

    4.       Knowledge Deployment - Knowledge deployment is the use of data mining within a target environment. In the deployment phase, insight and actionable information can be derived from data.

    -Big data is about data, plain and simple. Yes, you can add all sorts of adjectives when talking about “big” data, but at the end of the day, it’s all data.

    -Internet of Things is about data, devices, and connectivity. Data – big and small – is front and center in the Internet of Things world of connected devices.
    Good Business Analysts are

    • good communicators

    • problem-solvers,

    • critical thinkers.


      • They can create requirements specifications, analyze requirements, create visual models, facilitate elicitation sessions, and use the necessary business analyst tools. The ability to ask the right question to understand the problem inside out is also an important skill to be a good problem solver.
    Qualitative forecasting is subjective and it depends on the opinion of the customer. It is usually applied to the intermediate or long-range decisions. Quantitative forecasting depends on the past and present data to predict the future condition of the product. If there are no data available, or if the data available are not relevant to the forecasts, then qualitative methods must be used.
    The importance of Data Mining are:-
    -Discovery of Pattern
    -Prediction of key outcomes
    -Creation of actionable information(Data mining can derive actionable information from large volumes of data. For example, a town planner might use a model that predicts income based on demographics to develop a plan for low-income housing.)
    A propensity model is a statistical scorecard that is used to predict the behavior of your customer or prospect base.
    <br.Propensity models are often used to identify those most likely to respond to an offer or to focus retention activity on those most likely to churn.
    A business analyst is someone who analyzes an organization or business domain (real or hypothetical) and documents its business or processes or systems, assessing the business model or its integration with technology. It involves comparing facts and figures, listening to managers and shareholders to provide effective solutions for problems and the negotiation of issues between departments.
    Time series is a series of data points in which each data point is associated with a timestamp.

    Time series analysis comprises methods for analyzing time-series data in order to extract meaningful statistics and other characteristics of the data. Time series forecasting is the use of a model to predict future values based on previously observed values.
    Example – Rainfall in an area for a complete year, price of a stock in the stock market at different points of time in the day.
    UML stands for Unified Modeling Language. It is the standard in the industry for visualizing, documenting, and constructing various components of a system.
    A Business Analyst is a person who analyzes the working of an organization or domain and works on the quality of service provided by the same. He also assists in integrating and testing new solutions
    Few examples are

    - Web analytics (hits, time on website)

    - Retail( transaction history)

    - Manufacturing (Completion time)

    -Government (Tax collection)
    Be prepared for this question. A modeling language can be graphical or textual.

    -Graphical modeling languages use a diagram technique with named symbols that represent concepts and lines that connect the symbols and represent relationships and various other graphical notation to represent constraints.

    -Textual modeling languages may use standardized keywords accompanied by parameters or natural language terms and phrases to make computer-interpretable expressions. An example of a graphical modeling language and a corresponding textual modeling language is EXPRESS.
    You can select any presentation and showcase the effectiveness of the presentation. The presentation should have a story and you should be prepared to present it effectively(if needed)
    Though the answer is always subject to personal opinion, we can see what a general approach should be

    a)Food item combination can help you recommend food items, suppose most of the people orders curd with Paratha, then you can recommend curd to a person who orders Paratha(:P).

    b) A list of the most popular food item.

    c) Looking at the customer’s age and sex you can decide what to offer to a new customer. If the data suggests that a girl in the age group 20-25 likes chocolate ice cream then you can recommend this ice cream to the new girl customer.
    Python Flask is a micro-framework based on Python to develop a web application.

    It is a very simple application framework that has many extensions to build an enterprise level application.

    Flask does not provide a data abstraction layer or form validation by default. We can use external libraries on top of Flask to perform such tasks.
    Some of the main benefits of using Python are as follows:

    Easy to learn: Python is simple language. It is easy to learn for a new programmer.

    Large library: There is a large library for utilities in Python that can be used for different kinds of applications.

    Readability: Python has a variety of statements and expressions that are quite readable and very explicit in their use. It increases the readability of the overall code.

    Memory management: In Python, memory management is built into the Interpreter. So a developer does not have to spend effort on managing memory among objects.

    Complex built-in Data types: Python has built-in Complex data types like list, set, dict etc.

    These data types give a very good performance as well as save time in coding new features.
    A frozenset is a collection of unique values in Python. In addition to all the properties of set, a frozenset is immutable and washable.

    Once we have set the values in a frozenset, we cannot change. So we cannot use and update methods from a set on frozenset.

    Being hashable, we can use the objects in frozenset as keys in a Dictionary.
    EmpNo EName Job MGR HireDate Sal Comm DeptNo
    1234 Amit Waiter 8382 19-Oct-18 50000 500 50
    5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
    SELECT *
    FROM emp
    WHERE HireDate < (SELECT HireDate FROM emp WHERE EName = ‘Aman’)
    EmpNo EName Job MGR HireDate Sal Comm DeptNo
    1234 Amit Waiter 8382 19-Oct-18 50000 500 50
    5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
    SELECT *
    FROM emp e, dept d
    WHERE UPPER(d.loc) IN (‘DELHI,’BANGALORE’) AND e.DeptNo = d.DeptNo
    AND e.HireDate < (SELECT e.HireDate FROM emp e WHERE EName = ‘Aman’);
    EmpNo EName Job MGR HireDate Sal Comm DeptNo
    1234 Amit Waiter 8382 19-Oct-18 50000 500 50
    5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
    SELECT *
    FROM empSELECT *
    WHERE job in (SELECT job from emp WHERE EName IN (‘Aman’,’Amit’);
    EmpNo EName Job MGR HireDate Sal Comm DeptNo
    1234 Amit Waiter 8382 19-Oct-18 50000 500 50
    5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
    DeptNo Dname Loc
    50 Service Delhi
    51 Account Mumbai
    SELECT *
    FROM emp e, dept d
    WHERE UPPER(d.loc) IN (‘DELHI,’BANGALORE’) AND e.DeptNo = d.DeptNo
    AND e.HireDate < (SELECT e.HireDate FROM emp e WHERE EName = ‘Aman’);
    SELECT * FROM emp
    EmpNo EName Job MGR HireDate Sal Comm DeptNo
    1234 Amit Waiter 8382 19-Oct-18 50000 500 50
    5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
    WHERE TIMESTAMPDIFF(MONTH, HireDate, sysdate)
    EmpNo EName Job MGR HireDate Sal Comm DeptNo
    1234 Amit Waiter 8382 19-Oct-18 50000 500 50
    5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
    SELECT MAX(Sal)
    FROM emp;
    EmpNo EName Job MGR HireDate Sal Comm DeptNo
    1234 Amit Waiter 8382 19-Oct-18 50000 500 50
    5678 Ashish Analyst 8635 2-Nov-18 60000 200 51
    SELECT *
    FROM emp
    WHERE Salary = (SELECT MIN(Salary)
    FROM emp);
    setwd() and getwd() functions are used to set a working directory and get the working directory for R.
    setwd() is used to direct R to perform any action in that location and to directly import objects from there itself.
    getwd() is used to see which is the current working directory of R
    On the data window, navigate to the field. Right-click on the field, select Transform, and Split. The split will split the columns using the delimiter -. The split function will create two separate columns- EmpID and Name.
    Tableau provides different options and functions to manage dates effectively.

    -  Date property can be set by right-clicking on the data source in the Data section and select Date properties. Week Start day can be selected. Also, there is an option to specify a Fiscal Year Start Month. The Developer can also specify the Date Format to be followed in the design.

    How are dates handled in Tableau?



    - Dates can be Discreet or Continuous. The date level of a field can be changed by dropping it into the Row or Column shelf and selecting a pill drop menu. Using this menu, dates can be changed between discreet and continuous.



    How are dates handled in tableau?



    -  Sometimes Dates are not interpreted correctly by Tableau and dates - comes as String or some other format. In such cases, a field can be converted to Date in Tableau. On the data window, right-click on the field and select “Change Data Type”.

    -  Custom date format can be created by right-clicking on the date field and selecting Create Custom Date.

    -  Dates functions available in Tableau are

    how are dates handled in tableau?

    DATEDIFF('day',[Order Date],[Ship Date])
    Visualization is a pictorial representation of data.
    A primary key uniquely identifies a record in a table. The primary key in one table is referenced by a foreign key in another table. Two relational tables are linked based on the primary key and foreign key.
    The joins in SQL are:

    a) Inner Join. Returns matching rows from both the tables.
    b) Left/Outer join. Returns all rows from the left table and the matched rows from the right table.
    c) Right/Outer join. Returns all rows from the Right table and the matched rows from the left table.
    d) Full/Outer join. Returns all rows from table1 and from the table. It combines the result of both LEFT and RIGHT joins.
    e) Cartesian join. Join every row of one table to every row of another table.
    Union and Union All are used to combine the results of queries. Union eliminates duplicate records and Union All includes all the records.
    Data granularity refers to the level of detail or depth of data. It means the level at which data is stored in a fact table. For e.g. if data is stored at the Year level then it is at the lower granularity. If the data is stored at the Month or day level then it at higher granularity.
    Data transformation means transforming the data from its original format. Raw data may be in a different format than required by the report or dashboard. Data transformation is required to make the data more suited for the application. An example can be, removing time information from the DateTime field.
    Data can be discreet or continuous.
    Discreet data contains specific categories. For example, Regions - South and North or Years like 2011, 2012.
    Continuous data, on the other hand, is defined by a range and it can take any value over a continuous range.
    In Tableau, Dimensions are discreet and Measures are continuous.
    Dimensions are discrete and when dropped on the Color shelf displays different colors or color palette.
    Measures are continuous and when dropped on the Color shelf shows a gradient color.
    Workbooks can be published on Tableau Online or Tableau Server. In the absence of Tableau Server, Workbooks can also be viewed using Tableau Reader. Tableau reader accepts Tableau packaged workbook (twbx) which contains data.
    When Tableau Desktop is launched it provides an option to connect to the data.  Tableau can connect to a variety of data sources.

    -When connecting to data, Tableau provides an option to connect to data Live or as an Extract.

    -Data can be extracted to a .tde file. Extracts are faster than live connections.

    -When data is extracted, the symbol of the data source changes in the data window. It shows up with two cylinders.
    How to connect to data in Tableau?

    -Data Source can be saved as a TDS file or published to Tableau Server.

    Exercise: Use Tableau sample data source Sample- Superstore.xls located under \My Tableau Repository\Data sources.
    Connect to data live or as an extract.
    When connected as an extract, it will provide an option to save the data source as a TDE file. How to connect to data in Tableau?
    When a data source is created, Tableau automatically segregates data into Dimensions and Measure fields.

    - Dimension fields contain the textual attribute of the data. It provides the context to the Measure. Dimensions are generally used to create labels and filters. Dimensions are discreet and appear blue in color in the data pane and the view.
    - Measure fields contain the measurable attribute of the data - such as Revenue, Profit, or Population. Measures are continuous. They are axis in the charts and appear green color. Measures are analyzed by dimensions.
    Tableau provides an option to write custom queries in New Custom SQL. A pre-built query can be copied/pasted in the query editor. The Developer can also use this space to write a custom query. New Custom SQL option is available when connected to relational data sources.

     

    A query is created and tested in a SQL editor like TOAD or SQL Developer. How can you use this query in Tableau?

    Data source preparation also called data transformation is an important step in dashboarding. Data Source preparation depends on Business and visualization requirements. -

    - Review your source data to see if data is formatted correctly. If multiple tables or excel sheets are involved, look for the relationships between them. Create appropriate joins.
    - For successful visualization, data should have dimensions and measures.
    - If the source data contains a wrongly formatted Excel file, utilize Data Interpreter to format the file.
    - Use Split to segregate concatenated columns and use Pivot to transform Rows to Columns.
    - Data preparation may include renaming columns, creating hierarchies, Groups, and calculations.
    Data should be prepared so that it improves user experience and helps in the development of visualizations.

      -The Data source should be given a user-friendly name. Right-click on Data Source and select Rename.

      -Dimension and Measure columns should be given user-friendly names. Right-click on a field and select Rename.

      -the data types of the fields. If data types are incorrect then change the datatypes in Tableau.

      -Related fields should be organized in Folders. Right-click on a field and select Group By/ Folder.

      -Hierarchies help in drill-downs. Identify the columns to be included Hierarchy. Create Hierarchy by Shift or Ctrl select the columns to be included in Hierarchy, right-click, and select Hierarchy.

      -Provide Aliases to give a meaningful description of the values. For example, if the Region is SZ, you can alias it to South Zone.

      -Hide the columns, dimensions, and measures which are not used in the calculations or the view.

      -Create Calculations to achieve the required functionality. Name calculations in a user-friendly manner.

      -Apply appropriate filters to get meaning full data.

      -Use extract when possible. Extracts are faster than live connections.

      -Save the data source as a TDS file. This file can be shared and other developers can use this file for development.
    In the Dimension section, create a hierarchy for Category and Sub-Category.

    On a chart, a user wants to drill down from Category to Sub- Category. What data transformation should be applied to achieve this functionality?



    Exercise: Use Tableau sample data source Sample- Superstore.xls located under \My Tableau Repository\Data sources. Use the Orders datasheet and prepare data for dashboarding. Refer to downloaded workbook – Review Dimensions and Measures.
    Copy of a data source can be made by right-clicking on it and selecting Duplicate.

    How to create a copy of the data source?In what scenarios it will be useful?



    Copy of the data source can be useful when you have to create additional view/s of the same data source to create some visualizations like two separate charts for Year by Year comparisons.
    To ensure that all the developers are using the same data source, right-click on the data source and Publish on the Tableau Server.

    In a multi-developer environment, how do we make sure that all the developers are using the same datasource structure?



    While connecting to the data source, other developers can connect to the data source from the Tableau server option.


    In a multi-developer environment, how do we make sure that all the developers are using the same datasource structure?


    Developers using the published data source will be allowed to create their own calculations on the data source.

    Tableau Data Extract (TDE) file is a data extract or snapshot of the data. TDE files can be created by right-clicking on the data source and selecting Extract Data.

    -When an extract is created, the symbol of the data source icon changes to two cylinders.

    What is a TDE file and how it is created?



    -Extracts result in better performance as compared to live connections.

    -Extracts are useful in offline access to data. Extracts can be saved locally and do not require a live connection to the source data.

    -The extract is a snapshot of data, it needs to be refreshed as underlying data changes.

    -Filters can be applied while creating an extract.

    -While creating an extract data can be aggregated by dimensions.

    -An extract can be created for all the rows or incrementally.

    -Extract refreshes can be scheduled to run.

    What is a TDE file and how it is created?

    A TDS file stores the data source connection information. Developers can connect to the TDS file to get the data source details. TDS file is created by right-clicking on the data source and selecting “Add to Saved Data Sources”

    What is a TDS file?



    TDS file is in XML format and contains the following information
    • Data Source Type.
    • Data Source connection details such as Server, Port, Location of local files.
    • Joins, tables joined and type of join.
    • Groups
    • Sets
    • Calculated fields
    • Bins
    • Default field properties such as number formats, aggregation, and sort Order.

      Exercise: Use the file you have created in the previous section and save the data source .TDS file under \My Tableau Repository\Data sources. Navigate to the above path and open .Tds file and review it in a notepad.
    Data Blending is used when there are 2 independent data sources and visualization needs to merge the data from these data sources.

    -  In Data blending, one data source becomes primary and the second data source becomes secondary.


    - Data blending is not a join. Join occurs between the tables from the same data source.



    -To perform data blending there should be at least one common column between the data sources.

    - If a common field does not exist, then edit the relationship between the data sources. To edit relationships, navigate to the menu, and select Data/EditRelationships.

    - Data blending can be applied per visualization, it does not carry over to the entire workbook.
    The primary data source has a blue checkmark and secondary has an orange checkmark.

    How are primary and secondary datasources identified in data blending?

    A Join will be more appropriate in this scenario because the data source is the same.
    Tableau aggregates data automatically when the measure is placed on the view canvas. Default aggregation is Sum. But it can be changed to any other type of aggregation.
    This can be accomplished by creating an extract and choosing an incremental load option.

    A Source data table has 100,000 rows of data. Every month around 10,000 new rows get added to this table. What option you will choose to load only the new rows?

    Dimension filters are similar to the WHERE clause in the SQL query.
    Measure filter is similar to the HAVING clause.
    When a field is placed from the data window into the view canvas, Tableau creates a pill. Whether a pill is a dimension or measure or continuous or discreet affects the analysis and view.

    How does Pills work in Tableau?

    -The pill can contain continuous or discreet data. Continuous data can take any value in a range. Discreet data have distinct separate values.

    -Dimensions are discreet and Measures are continuous.

    -Discreet pills are blue in color and continuous are green in color.

    -Dimension and Measure can be changed to discreet or continuous by right-clicking and selecting the desired type.

    -When a continuous pill is dropped into the view it creates an axis.

    -When a discreet pill is dropped into the view, it creates a label/header.

    -Color is also decided to depend on whether - the pill is continuous or discreet.

    -Filtering on a discreet field such as category will ask for the specific values.

    -Filtering the continuous field such as profit will first ask if you want to filter at the row-level or

    -Aggregate level and then bring up options for continuous ranges.


    -In Maps, the default color depends on whether a field is continuous or discreet.
    -A measure on Color will create a filled map. A dimension on the color will create a Symbol map. But whether the color is gradient or palette still depends on whether the pill/field is continuous or discreet.

    Exercise: Use Tableau sample data source Sample- Superstore.xls located under \My Tableau Repository\Data sources.
    Create a Map by using State and Region in one sheet and State and Sales in another sheet.
    Refer to downloaded workbook – sheet/s - SymbolMap_Color and FilledMap_Color.
    It is important to structure your workbook for ease of use and development.

    -Fields can be organized in folders.
    -Keep related fields in a folder.
    -Hide all unused fields.
    -Workbook many contain multiple sheets and dashboards. Color related sheets and dashboards.
    -Hide unused worksheets.
    Datasource can be replaced by following the below steps:

    - Add a new data source to Tableau. Navigate to the main menu and select Data/New Data Source

    Q46. How to change/replace the data source after developing visualizations or dashboard?



    - Make sure the new data source contains the same data structure in terms of calculations, hierarchies, parameters, etc.

    - Right-click on the old data source and select Replace data.

    How to change/replace the data source after developing visualizations or dashboard?

    -

    - Select the Current and Replacement data sources

    How to change/replace the data source after developing visualizations or dashboard?

    - After replacing the Datasource, close the original data source.
    Tableau generates few fields automatically in the data window. These fields get generated when you connect to the data and create a sheet. These fields are Measure Names, Measure Values, and Number of Records.
    Sometimes the source data is not in the right format. Headers and rows are not stored correctly in the excel file. In such scenarios, Tableau invokes a “Data interpreter”.

    The data interpreter option is invoked automatically when Tableau recognizes that data is not in the right format.What is Data interpreter?

    Data Interpreter formats the data correctly and also provides you the option to Pivot the columns into rows.

    What is Data interpreter?

    Calculations or calculated field/s help in enhancing your visualization and implementing business rules. Calculated fields are created by using different Tableau functions.

    Types of calculations available are:

    a) Regular Calculations. This calculation is sent to the data source for processing and the result is returned to Tableau.

    b) Table Calculation. Calculation occurs on top of the returned result set or chart.

    c)Quick Table Calculation. These calculations are predefined Table calculations provided by Tableau.

    d) Level of Detail (LOD) calculation. This calculation computes aggregation that is out of the level of detail of the view.
    The regular calculation can be created by right-clicking on the white space in the data section and select “Create Calculated Field”.

    How to create a Regular calculation?



    A calculated field can also be created by navigating to the Menu/Analysis and selecting Create Calculated Field.

     

    A calculated field can use any of the Tableau defined functions such as



    How to create a Regular calculation?  


    Exercise: Use Tableau sample data source Sample- Superstore.xls located under \My Tableau Repository\Data sources. Use the Orders Datasheet. Create a calculation for Sales with Discount. Use the formula:

    Sales * Discount.

    Since Table calculation works on the returned result set or the chart, it can be created on the view canvas.

    Create a visualization by placing dimensions and measures on the view canvas. Click on the measure pill and select Add Table Calculation.

     

    How is Table calculation created?



    Table calculation can also be created just like regular calculation but will use the Table calculation functions.

    How is Table calculation created?



    Exercise: Use Tableau sample data source Sample- Superstore.xls located under \My Tableau Repository\Data sources.
    Use the Orders datasheet. Place State on Rows and Sales on the Columns Shelf. Click on the Pill to add Table Calculation for Rank.
    Tableau provides some pre-defined Table calculations called Quick Table calculations.

    Quick Table calculations also work on returned result set and are created by right-clicking on the measure pill on the view canvas and selecting “Quick Table Calculation” The types of quick calculations available are

    What are Quick Table calculations in Tableau?

    Tableau internally assigns all fields in the dataset to be addressing or partitioning. Partitioning field partitions data into separate individual sections. Calculations are applied to these sections.

    Addressing field provides the direction in which the calculation will take place.

    Exercise: Use Tableau sample data source Sample- Superstore.xls located under \My Tableau Repository\Data sources.
    Use the Orders datasheet.
    Place Order Date on the Rows, drill down to Quarters. Double click on Sales. Create a Table calculation for “Running Total”.  After the visualization is created, navigate to Sum(Sales) and select edit Table calculation.
    Under Running Along, choose the option for Advanced and see which columns are used for Partitioning and Addressing. Switch the columns around to see the changes in the output. .
    Refer to the Partitioning field’s definition in the previous question.
    First, Last and Index Table calculation functions provide information about the position of a data element within a partition.

    First() – returns the number of rows from the current row to the first row in the partition. It is a negative number.

    Last () – returns the number of rows from the current row to the last row in the partition. It is a positive number.

    Index() – returns the index of the current row in the partition. The first-row index starts with

    1. Exercise: Create an example to understand First, Last and Index functions.
    Place Category on the Rows and drill down to Sub-Category. Place Sales on the Columns. Create calculations for First(), Last(), and Index() and place them on the table in the view canvas.
    Observe the output.
    Navigate to the function pills on the left under Marks and use Compute using to change the partitioning fields and see the difference in your output.

    What is First (), Last () and Index () functions in Tableau?

    LOD calculations compute aggregation that is outside the “level of detail” of the view.

    LOD expression has a specific syntax and works with keywords such as Include, Exclude, and Fixed. LOD calculation follows the below syntax

    {Keyword [Dim1],[Dim2]: Sum(Measure field)}.
    LOD expressions have the following keywords

    a) Fixed computes the value using the mentioned dimensions ignoring the dimension in the view

    b) Include computes the value using the mentioned dimension in addition to the dimensions in the view

    c) Exclude will ignore the mentioned dimension. It ignores the dimension even if it is used in the view.

    d) If no dimension or Keyword is specified then the expression is “Table scoped” which is the complete aggregate of the data. {Sum(Sales)}

    e) Only field names can be used in the Dimension declaration. Calculated fields can be used but not the calculation expression. Sets, parameters, or Table calculations are not allowed.

    f) The LOD expressions that use Exclude or Include always result in measures. These results cannot be binned.

    g) Fixed can be an expression or a dimension based on the aggregation used. If the aggregate expression results in a measure then the overall expression will be a measure. If aggregation such as string, Boolean, or Date types results in dimension then the overall expression will be a dimension. The fixed expression can be converted to a dimension or a measure.

    h) Numeric Fixed LOD expressions can be binned unless the aggregate expression is a date.
    LOD expressions are used to aggregate outside the level of detail of the view. In this case, the level of detail is defined by ZipCode. So if a developer wants to calculate the Sum by Profit at the State level, he can create a calculated field with the following LOD expression and place this calculated field in the Detail shelf.

    {Fixed [State]: SUM ([Profit])}

    This expression will compute at the State level. It will ignore the dimension in the view i.e. ZipCode.
    Attr checks to see if there is only one value for a given field for all rows in a result set. Attr is used to aggregate the dimension.

    Whenever we need to check for a single value such as if State = “NJ”, use Attr
    When using Dimension field with a measure field in a calculation, you get the following error. What is the resolution of the error?

    This error occurs when a dimension is used with an aggregated function. If the dimension returns more than one row of data, this error occurs. This error can be resolved by using the ATTR function with the dimension
    Tableau is a Business Intelligence Application. Tableau is used to create visualizations and interpret data. Tableau follows the basic principles of BI i.e. extracting raw data, transforming it, and presenting the data visually so that Business users can make informed decisions.
    Tableau Architecture follows as below

    Explain Tableau Architecture?

    1)Data.Tableau can connect to any format of source data.
    2)Data Connectors. Tableau provides over 40 optimized data connectors. To various data sources such as MS Excel, MS SQL Server, Google Big Query, Amazon RedShift, Oracle, and others.
    It also provides a generic ODBC connector for systems without a native connector.
    Data can be used in-memory or live.

    3)Components. The following components handle the server operations
    a)Application server. Application Server processes (wgserver.exe) handle content browsing, Server- Administration, and authentication to Tableau server web and mobile interfaces.
    b)VizQL Server. When a user/client requests a visualization, it sends a request to the VizQL process (Vizqlserver.exe). The VizQL process in turn sends queries to the data source, returning a result set in the form of images. 
    c)Data Server. It facilitates the management of data sources on the server.

    4)Gateway/Load balancer. Gateway directs requests to other components.
    5)View. Users can view Tableau dashboards thru Tableau desktop or via zero-footprint HTML 5 in a web or mobile browser.
    The basic components of Tableau are Tableau Desktop, Tableau Reader, and Tableau Server.

    -The developer uses Tableau desktop to create visualizations, dashboards, and stories.

    -Dashboards can be deployed on the Tableau Server. Users can access dashboards stored on the server through the server URL. Tableau desktop can use data as a “Live” connection or as an “Extract” (TDE).

    -Data connection/s used in developing dashboards is reusable. It can be published to the server. Data load schedules can be created to meet user requirements.

    -Tableau desktop design file is called a “Workbook”. It has an extension .twb.

    -Tableau workbooks can also be packaged with data. This packaged workbook is a zip file with extension .twbx.

    -In the absence of the server, Tableau design files can be viewed by using Tableau Reader. Tableau reader is also a free download and can open twbx files.
    Tableau using powerful visualization helps in understanding the data.

    Tableau dashboards are interactive and display the whole picture of the data. All the data is present, data analysis across different time periods and dimensions is faster.

    Datasource used in the dashboard design can be shared with other developers and users, this helps in maintaining the single version of the truth.

    Development is faster as compared to other applications. Tableau provides easy to use functionalities to create data hierarchy, calculated fields, filters, parameters, Sets, and Bins.

    Development is not IT-centric. It gives power to business users. Users can design their own dashboards.

    Creating worksheets, dashboards, and stories are easy.

    Based on the data, Tableau’s “Show Me” feature suggests users the best visualization type to use. This helps novice users in creating meaningful charts and tables.

    Tableau can handle huge amounts of data. Connection to the data can be live or to an extract.

    For a quick review, workbooks can be emailed as packaged workbooks with data.

    Publishing to the server and applying security is easy.
    A Tableau professional’s responsibilities may differ based on specific organization but at high-level it consists of

    -Und<brerstanding business requirements.

    -Analyzing data sources and relationships.

    -Extracting data from the required data sources. Making decisions in collaboration with Business users on connecting data live or as an extract.

    -Applying required data transformations, creating calculations, Sets, Bins as driven by the requirements.

    -Creating visualizations that help in answering business questions. These visualizations are created and displayed in Sheets, dashboards, and stories.

    -Deploying the dashboards on the server
    BI dashboards are similar to the car dashboard.

    The car dashboard provides the complete status of the working of a car, such as Gas, mileage, Air pressure, and so on.

    BI Dashboard provides a snapshot of the organization or Key Performance Indicator (KPI).
    Dashboards in Tableau are created by either clicking on the Dashboard icon at the bottom of the screen

    How are Dashboards are in Tableau?



    - or from the Menu/ Dashboard.

    - Dashboards contain one or multiple sheets.

    - Actions provide interactivity to the dashboards.

    - Filters present in the individual sheets are to dashboards too. These filters can be applied to a specific sheet or all sheets using the data source.

    - The dashboard contains “Dashboard Objects”. These objects are available in the left pane. These dashboard objects are layout containers – horizontal or vertical. These objects can be used to add Image, webpage, text, and Blank container. Use these objects to enhance your dashboard.

    - Dashboard size and sheet coordinates can also be specified.

    How are Dashboards are in Tableau?



    Exercise: Create a Dashboard using Sheets, FilledMap_color, Combo chart, Parameters, and TableCalculation_Rank.  
    Interactivity between dashboards is achieved by the use of “Actions”.
    Actions provide interactivity to the dashboards. Actions can be of different types

    Filter Actions. This type of Action is used to create interaction between the sheets in a dashboard.

    Highlight Actions. It is used to color the selected Marks on a chart.

    URL Actions. This type of Action is used to create a hyperlink to another webpage.

    Actions can be configured by navigating to Menu – Dashboard/Actions.
    Best practices should be followed to keep the design clean. Data should be organized and visualization should be easy to understand.

    - Use folders to organize related fields or calculations.

    - Name the data connections appropriately.

    - Use standard naming conventions for the calculated fields.

    - Use a standard naming convention for the parameters.

    - Use standard naming conventions to name the sheets and dashboards.
    Tableau provides different kinds of visualization objects. Show Me option shows all the charts available. Based on selected dimensions and measures, Tableau shows the suggested charts in “Show Me”

    What are the charts types available in Tableau?

    Most of the charts require one or more dimensions and one or more Measures/expressions.
    The relationship between the two measures is displayed using a Scatter chart.
    Combo chart or combination chart should be used to display 2 measures by a dimension.

    Place Region on the Columns Shelf. Drop Sales and Profit on the Rows shelf.

    Right-click on the pill of Profit and select Dual Axis.

    Click on Sum(Sales) pill and change the Mark Type to Bar

    Click on Sum(Profit) pill and change the Mark Type to Line.

    Exercise: Use Orders datasheet. Create the above combo chart for Sales and Profit by Region.
    Filters help in restricting or excluding data. Filters are applied independently of each other. Each time a filter is applied a separate query is sent to the data source. Exceptions to the rule are the “Context filters”. Tableau provides different types of filters.

    Data Source filters. These filters are added to the data source. These filters restrict whole data.

    What are Filters? How many types of Filters are there in Tableau?

    Extract filters. These filters are implemented while creating an extract. Right-click on the data source and select Extract data. In the extract data options, select the filter/s.

    What are Filters? How many types of Filters are there in Tableau?



    Context filters. These filters create a subset of data. Context filter, if applied, acts like a temp table. All the other filters act on this sub-set of data.

    Quick filters. These filters give the user an option to select a filter while viewing a dashboard or a visualization. Place a field on the Filters shelf, right-click and select “Show filter”

    What are Filters? How many types of Filters are there in Tableau?
    Usually, Dimensions are discreet and Measures are continuous. When a Dimension field is used as a filter, it displays as separate categories in a List or a Drop-down. If Measure field is used as a filter, it gets displayed as a slider.


    - Parameters provide a dynamic value to the hardcoded or constant value in a visualization.

    - Parameters are single select and provide context only when used in a calculated field.

    - Parameters can work as a data input to the visualization.

    - A good use of parameter will be, for example, if you want to change the measure in a chart based on the values selected by a user.

    - Parameters are displayed at the bottom of the data section.
    Parameter is not a filter but can work with a filter.
    This can be done by creating a string parameter, say “Param select Measure” for these measures.

    Use this parameter in a calculated field, say, “Select Measure” Create a visualization using this calculated field Select Measure.

    Display the “Param Select Measure” for user selection.

    Exercise: Create a new sheet, use Region, Sales, Profit, and Quantity.
    Create a parameter, so that the user can pick a measure from the list and it should change the measure in the visualization.
    Sorting means arranging data in ascending, descending, or customized order.

    - Sorting can be ascending or descending. Sorting can be on dimensions or measures.

    - Sorting options can be selected from the toolbar.

    In how many ways can be Sorting be done in Tableau?

    Sorting option is also available when your mouse hovers over the axis of a chart.

    - Pill sorting option is available only for dimensions.

    - Sorting can also be performed by dragging and dropping data elements on a chart.

    In how many ways can be Sorting be done in Tableau?



    - Color legends also can be sorted by manually.
    -Maps are created for the geographical or location fields.

    -Tableau automatically identifies geographical fields like County, State, City and zip-codes. It assigns Latitude and Longitude to each value of the field based on the data present in Tableau Map-Server.

    -Fields having the geographical role assigned can be identified by a globe symbol next to them.

    -Sometimes Tableau cannot recognize a field as a geographical location. For example, sometimes Zip code would be identified as a number. In such instances, you can right-click and assign a geographical role to the field.

    -If Tableau does not contain the desired Latitudes and Longitudes, you can define them in a database or a CSV file.

    Geographical roles available in Tableau are

    How are Maps created in Tableau?



    -Area code contains USA telephone area codes.

    -CBSA/MSA denotes USA metropolitan Statistical Areas.

    - City represents worldwide cities with a a population of more than 15,000.

    -Congressional District represents U.S. congressional districts.

    -County/Region represents worldwide countries.

    -County represents counties of few countries such as the USA, France, German, etc.

    -State/Province represents worldwide states.

    -Zipcode/ Postal code of selected countries are available

    Exercise: Use Tableau sample data source Sample- Superstore.xls located under \My Tableau Repository\Data sources.
    Use the Orders datasheet. Place State on Rows and Sales on the Column Shelf. This will create a Map that will give Sales for each State.
    “146 Unknown” means Tableau is unable to plot some 146 cities. Some of the cities may also be duplicated.

    In such a scenario, drop State on the Detail shelf.

    Alternatively, you can click on “146 Unknown” and from the popup dialogue box select “Edit Locations”

    When a developer uses City and Sales to generate a Map, “146 unknown” is displayed at the bottom of the Map. How can this issue be resolved

    In Tableau, the granularity of data is defined by the Dimensions field.

    The dimension field dropped in the Detail shelf will change the granularity of the visualization.

    Dropping measures in the Detail shelf will have no effect.
    Tooltip is a shelf next to Detail.

    - The tooltip displays the data description when the user hovers over the data elements of the chart.

    - The tooltip displays the data elements that are in the view canvas. You can display other data elements by placing them on the Detail. Use those data elements that do not change the granularity of the data.

    - Tool Tip can be formatted to add different colors and fonts.

    - Any comments can also be displayed in the ToolTip.

    The screenshot is given below shows Tooltip when the user hovers over a state in the Map.

    What is ToolTip and how to provide a ToolTip?

    Complete all the formatting changes on one sheet, say, Source Sheet. Right-click on the sheet and select Copy Formatting.

    Go to the Target Sheet or other sheets - where you want to copy the formatting, right-click on the sheet and select “Paste formatting”.

    How to make sure that all the sheets have the same look and feel?

    Fields can be combined into a new dimension by creating a Group.

    Groups are used to combine different dimension values. Once grouped, these values show up as a new dimension field under Dimensions. Group is also used to improve data quality in a dashboard. For eg., if visualization contains values such as Unites States, USA, US, you can combine all these values into one Group and have a value like the USA.
    A Group can be created from the view canvas such as from Table, Bar chart, or Scatter chart.

    On a Bar chart, select the headers of a dimension and click on the “pin” icon.

    a scatter chart, highlight the marks on the chart and click on the “pin” icon to the group.


    Group can also be created in the data window. Groups can also be calculated by creating a calculated field.

    How is a Group created?



    Exercise: Use Tableau sample data source Sample- Superstore.xls Use Orders datasheet. Create a Group called Office Supplier – Group.
    Bins are used to converting a continuous measure or dimension into discreet buckets.

    Bins, when created, display as a field under Dimensions.

    Bins are created by right-clicking on a measure field and selecting Create/Bins.  Tableau suggests the size of the Bin or it can be entered by the developer.

    What are Bins in Tableau?



    Bins can also be created by using a calculated field.
    In Tableau, measures or dimensions are colored by placing them on the Color Shelf.

    -The color of a data element is decided by it being continuous or discreet.

    -A discreet field on the Color shelf produces a “color palette”. A continuous field creates a “gradient”.

    -Click on the color shelf to Edit Colors. To display consistent color, double click on the color and provide HTML code or RGB values.
    A Custom color palette can be provided to Tableau by modifying the Preferences.tps file. It is an XML file and is located under the “My Tableau Repository” folder on your machine or where Tableau Desktop is installed.

    If no custom color is specified, this file appears with place holder such as

    -?xml version='1.0'?>

    -workbook>

    -/workbook>

    Custom color can be specified in this file by using the HTML tags for Color Type and hexadecimal value.
    Sets are used to create sub-set of the data based on a condition such as Top 10 Customers by sales.

    -Sets can be created manually from the visualization in view canvas. On a scatter graph, highlight the marks which you want to use to create a Set, and select Create Set by clicking on the overlapping circle icon from the pop-up toolbar.

    -Sets can also be created dynamically from the data section. Right-click on any dimension and select Create/Sets. On the dialogue box, navigate to Top

    A developer wants to create a “subset of data” such as “Top 10 customers by sales”. This subset of data can be used as a filter in other sheets. What visualization technique should be used in such a scenario?



    -Sets can be used with other sets. Sets can be used as a filter. They can also be used in a calculated field.

    -Sets once created, show up in the data section below Measures.

    -Sets are available in all the sheets and can be used in multiple visualizations.

    -Exercise: Use Tableau sample data source Sample- Superstore.xls Use Orders datasheet. Create a SET for Top 10 Customers by Sales.
    Sets and filters are similar with few differences.

    - A set can be used as a filter. Filters are available only in one sheet but Sets are available as a part of data and therefore available in any sheet you create.

    - Since Set becomes a part of the data, it is available when the data source is saved as a.TDS file and shared across other dashboards.

    - Sets get created automatically when Action is used in a dashboard.

    - Sets get created, when User filter is applied to the dashboard for security.
    Reason: It assumes that the input variable is independent, but in real world it is unrealistic, since all the features would be dependent on each other.
    A. Mean is affected badly by the outliers.
    It’s said that if a Billionaire walks into a cheap bar, the average crowd becomes a millionaire
    A. Whenever there is a data point 0, then you cannot compute the Harmonic mean
    Like zero, it is impossible to calculate Geometric Mean with negative numbers. However, there are several work-around for this problem, all of which require that the negative values be converted or transformed to a meaningful positive equivalent value.
    A. Median is the 50th percentile which is equal to 5th decile, each decile denotes 10 percentile
    A. Bi-modal
    A. A series can have more than one value for mode. Example – 2,3,4,5,5,5,4,4,6,7,7 Here both 5 and 4 are mode
    A. The measure of dispersion can never be negative. It is never negative since every term in the variance sum is squared and therefore either positive or zero. It has squared units.
    A. Mean is generally taken as the center of distribution. So, the answer is c
    A. Discrete and continuous are the two types of a random variables. A percentile is definitely a quantile. R can approximate quantiles for common distribution
    Ans. The sum of the percent frequencies will always be 100
    Correlation can tell you something about the relationship between variables. It is used to understand:
    1. Whether the relationship is positive or negative
    2. The strength of the relationship.
    In statistical significance testing, the p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. If the p-value is less than 0.05 or 0.01, corresponding respectively to a 5% or 1% chance of rejecting the null hypothesis when it is true (Type I error).
    When you roll two dice, you have 6 possibilities for each roll (6 sides). This is 36 total combinations.
    Let's list the combinations that result in sums greater than 9.
    (4,6) (6,4) (5,5) (6,5) (5,6) (6,6)
    That's 6 out of the 36 total possibilities. Therefore, the remaining 30/36 possibilities fulfill the less than or equal to 9 requirements. Simplifying by a factor of 6, that's 5/6 chance.
    Shorter. Regression to the mean
    Collaborative Filtering
    The goodness of fit measure. Variance explained by the regression / total variance Remember, the more predictors you add the higher R^2 becomes. Hence use adjusted R^2 which adjusts for the degrees of freedom or train error metrics
    Statistically, It depends on the quality of your data, for example, if your data is biased, just getting more data won’t help. It depends on your model. If your model suffers from high bias, getting more data won’t improve your test results beyond a point. You’d need to add more features, etc.
    Practically, Also there’s a tradeoff between having more data and the additional storage, computational power, memory it requires. Hence, always think about the cost of having more data.
    Multicollinearity refers to a situation in which two or more explanatory variables in a multiple regression model are highly linearly related.
    Leave the model as is, despite multicollinearity. The presence of multicollinearity doesn’t affect the efficiency of extrapolating the fitted model to new data provided that the predictor variables follow the same pattern of multicollinearity in the new data as in the data on which the regression model is based.
    principal component regression
    There are many tests, few are:-
    -A/B Test
    -Student’s T Test
    -Chi Square Test
    -Fisher’s Exact Test
    -Mann-Whitney Test
    In Tableau, visualization is created on a Sheet.

    - To create any visualization, the developer needs to drop Dimension and/or Measure on the view canvas.

    - When you drop a measure in the view canvas, Tableau automatically aggregates the data. Default aggregation is Sum. These aggregations can be changed to other types.

    - Show Me feature suggests the best visualization for the number of Dimensions and Measures selected.

    - Dimensions or measures can be used to create filters and quick filters. These filters or quick filters help in providing context to the data.

    - Visualization can have Color, calculated field, Shape, and Size.

    -One or multiple Sheets are combined to create a Dashboard.

    -Sheets and Dashboards are used to create a Story.

    How is visualization created in Tableau?

    SELECT deptno, count(deptno)
    FROM emp
    GROUP BY deptno
    HAVING count(*)>3;
    Some of the important ways of preventing overfitting are as follows:

     a) Cross Validation: One of the best ways to prevent overfitting is the technique of Cross Validation.

    In the simplest version of this technique, we divide the dataset into two populations.
    One is a training population and the other is the testing population.

    By using Cross-Validation, we use the training population to create the model, whereas the testing population is used for testing the model. Once we have the correct model, it works well with training as well as the test population.
    Further, there are many versions of cross-validation by performing test/train split in different ways. One popular method is called K-fold cross-validation.

    b) Collect more Data: We can also avoid overfitting by collecting more data in the first place. When we have a small dataset, the chances of overfitting increase. We can collect more data to build the correct model.

    c) Reduce the number of features: In a complex model with the problem of overfitting, it is better to reduce the number of features that are not strong predictors. This helps in reducing the variance as well as boosting the performance of the model.

    d) Stopping Early: When we iterate multiple times to get a model, it can memorize the data. In such a scenario overfitting occurs. To prevent this situation, we can stop early in the iterations. This helps in stopping the training process at an optimum point.
    The problems in the real world are of a large variety and type. Each problem requires its own analysis. It can be solved in multiple ways. Some of the approaches to solve a problem by implementing Machine Learning are as follows:

    • a) Inductive learning

    • b) Unsupervised learning

    • c) Recommendation system

    • d) Predictive modeling

    • e) Artificial Neural Networks

    • f) Decision tree learning

    • g) Deep learning

    • h) Support vector machine

    • i) Clustering

    • j) Bayesian networks
    There are mainly two types of Decision Trees in Data Mining.

    a. Classification Tree: In a classification tree, class labels are represented by leaves in the tree. The branches of the tree represent the conjunction of features. There are distinct classes in a classification tree,

    b. Regression Tree: In a regression tree, the target variable can take continuous values. It means the result can be a wide range of values. The regression tree is built by binary recursive partitioning.
    Some of the popular algorithms of Machine Learning are as follows:

    • a) Linear Regression

    • b) Logistic Regression

    • c) K Nearest Neighbor (KNN)

    • d) Decision Tree Learning

    • e) Artificial Neural Network

    • f) Gradient Boosting Algorithm

    • g) Naive Bayes Algorithm

    • h) Support Vector Machine (SVM)
    By utilization of shared resources and infrastructure, the delivery of computing as a service is called Cloud Computing. This service is often rendered via a network.
    The concept of Cloud Computing can be used to provide computational, data access, storage and software services. These services can be accessed/utilized by the end-user over the internet.
    When we talk about the end-user, we are usually referring to the person accessing the cloud from a remote location. Any person/user utilizing the services offered in a cloud as a part of a financial agreement is known as the end-user.
    There are several ways by which an end-user can access services present at a particular cloud. Some of these require the user to plug into the cloud via:

    a) Web Browser

    b) Cloud based application

    c) Mobile applications

    d) Business software
    When multiple IT components are packaged into a single, user-based solution; the offered package is known as converged infrastructure. Such an infrastructure may include services like data storage, infrastructure management, orchestration, servers, networking equipment and automation, all bundled into one optimized solution for the end user.
    The three fundamental models of Cloud computing include:

    a) Infrastructure as a service (IaaS)

    b) Platform as a service (PaaS)

    c) Software as a service (SaaS)
    PaaS (Platform as a service) is a cloud model in which computing platform is offered as a service rather than equipment. This means that an OS (Operating system), Database, language execution environment or/and a web server is offered over a network environment (Typically, the internet).
    SaaS (Software as a service) is a cloud model in which software is remotely offered as a service. This means that rather than installing a typical software onto your system, you can access it and use it using a web browser or a cloud-based application.
    IaaS (Infrastructure as a service) is a cloud model in which computer itself is delivered as a service. This means that Cloud service providers can offer you virtual machines, networks, firewalls and raw storage blocks. IaaS providers usually bill the end user based on utility computing basis (amount of resources utilized).
    Service providers may offer access to their cloud via Laptops, tablets, smartphones and desktops. Now, some of the cloud service providers are offering their services via thin clients and the Chrome-book.
    Different deployment models of cloud computing include:

    a) Public Cloud – Standard cloud computing model.

    b) Community Cloud – Infrastructure shared between several organizations.

    c) Hybrid Cloud – Composed of multiple clouds bound together to provide services.

    d) Private Cloud – Clouds intended for a single organization only.
    Internet-based email accounts have been utilized since the emergence and popularization of the internet. Internet-based email accounts are another cloud service that is offered. One of the pioneers includes Hotmail, Yahoo and MSN. This way Internet-based email can be regarded as one of the first cloud services because you have a fully functioning email account hosted on an external server.
    Some of the potential concerns shared by users when going for a cloud storage service include:

    a) Security of stored data.

    b) Availability and reliability of offered service.

    c) Transfer speeds and general performance.
    A cloud computing service can be thought of as consisting of three pyramid layers. These layers are:

    a) Cloud Infrastructure

    b) Cloud Platform

    c) Cloud Application
    A cloud is a blend of hardware, storage, networks, interfaces and services that aid in delivering computing as a service. A cloud consists of three users:end-user, cloud service provider and business management user.
    The basic four characteristics of cloud computing are:

    a) Scalability and elasticity – Increase or decrease services on demand

    b) Self-service provisioning – Help yourself in major problematic scenarios without waiting for the response of your provider

    c) Pay-per-usage billing model – Pay for what you use

    d) Standardized interfaces – Interfaces applicable to all major models
    Some important features are as follows:

    a) Accessing and managing software

    b) Serve several clients at a given time

    c) Centralizing all activities in a Web environment

    d) Centralize the updating feature of the software
    Scalability is the provision of expanding or reducing the extent of services you require, depending on the situation. This means that if you require more of a service, you only have to ask your service provider and he will provide you with that additional service for an additional charge.
    a) Users don’t actually ‘own’ the data, service provider does.

    b) Excessive dependability on the service provider.

    c) Data disaster recovery can be a major concern.

    d) There are many problems associated with data migration if you change your service provider.

    e) What happens if the provider goes out of business?
    a) You get access to a big range of applications without having to actually own them

    b) Advantage of mobility – Excess anything, anywhere

    c) Cost-efficient

    d) Effective resource sharing

    e) Pay for what you use, rather than paying for what you own

    f) On-demand scalability at any instance
    Technically speaking, all applications can be run by a cloud but it is always a good idea not to put system dedicated applications (that run on minimum latency) on the cloud. Good examples include Windows default Disk Defragmentation Tool.
    Some big concerns include:

    a) Secure transfer of your data – No external attacks while the transfer of data from/to the cloud.

    b) Location of your data – Geographical location of data.

    c) Control of your data – Who has access to your data?
    Cluster is defined as a large group of computers that merge their capabilities and work as one. This aggregating of computational power is also called clustering of services. Each cluster is then referred to a single node in computing terms.
    A data center can be thought of as a collection of servers which hosts applications and storage space. Physically, a data center can be a big room full of servers that can be accessed from anywhere in the world via the internet.
    The two biggest concerns for a Mobile client are:

    a) Security of data

    b) Speed of data transmission
    This model has the following advantages:

    a) There is a large audience for this model.

    b) It is highly fault-tolerant.

    c) HTTPS-encrypted access provides privacy.

    d) Many provider options are available.

    e) It is cost-effective.
    a) Lack of end-to-end QOS (quality of service)

    b) Service-level agreements (SLAs) are difficult to reach

    c) Latency problems are widespread

    d) Downtime is out of your control
    There are two types of hypervisors:

    a) Bare-metal Hypervisor – runs directly on hardware.

    b)Virtualized Hypervisors – run on a virtualized platform.
    a) Basic public internet

    b) Accelerated internet

    c) Optimized overlay

    d) Site-to-site VPN
    A Cloud Storm occurs when multiple cloud computing environments are attached collectively. A Cloud Storm can also be regarded as a Cloud Network.
    a) The Internet bandwidth between the cloud and your organization

    b) The round-trip time between the cloud and your organization

    c) The actual response time of the cloud
    When connecting to the cloud, it is important to determine if your connection type is symmetric or asymmetric. In a symmetric connection, sending and receiving data rate is the same. In an asymmetric connection, upload speed to the cloud is usually slower than the download speed.
    The geographical distance between the source of the data and the client using it is usually quite large. This means that there is always a lag time between the transmission of data between the client and the service provider. This makes instantaneous operations very problematic. Good examples would be Cloud providers hosting high-end graphics games. These would be hard to play owing to the latency issues as discussed since there will always be a delay between ‘Button pressed at user’s end’ and ‘the action taken in the game itself.
    The following factors are important as applied to SaaS:

    a) Time to value

    b) Trial period – How long is the trial period of the service?

    c) Low entry costs

    d) Offered Service – What is the level of service required?

    e) Total Investment – How much investment is required in your choice of Cloud?

    f) Security – How secure is your data?

    g) Reduced Capital Expense – How much you save when you switch to another service?
    Mobile clients run the service from their laptops to PDAs and smartphones, e.g. BlackBerry or iPhone. Although some applications are not supported by such a setup as the popularity of mobile devices is on the high, most of the providers are turning their heads to Mobile solutions for Cloud Computing. Mobile cloud clients do not access the cloud from their systems as their choice resides with gadgets they can use ‘on the go’.
    SaaS can be divided into two major categories:

    a) Customer-oriented services – Offered to the general public on a subscription

    b) Line of business services – Offered to enterprises and companies via subscription
    Some of the potential concerns shared by users when going for a cloud storage service include:

    a) Security of stored data.

    b) Availability and reliability of offered service.

    c) Transfer speeds and general performance.
    We use Linear Regression to determine the price of cars, houses, profit etc. This is a technique to estimate the real value based on the variables of continuous range.

    E.g. the price of a house can range from $5000 to $1,500,000 based on the square ft. area, year of making, size of the lot, etc. It can be any number between this range.

    One of the simple ways of performing Linear Regression is by drawing a straight line. Once we have the best fit line based on our training data, we can use it for predicting the value of actual data.

    In a straight line we have to estimate the slope a and b intercept. The equation is:

    y = aX + b

    Let say we have a graph with X = sq. ft. of house and y = price of the house. Our model can estimate and give us a = 100 and b = 190 for the best fit line.

    Now we can get the value of any house by using following formula:

    y = 100X + 190

    We just need to replace X with the sq. ft. of the house to get the price of the house.

    In Linear regression X is the independent variable and y is the dependent variable. The value of y is dependent on the value of X.
    Logistic Regression is a classification algorithm. Although from the name it appears to be regression technique, it is used for classification problems.

    We can use it to determine the class of data. It can be a binary class like 0,1 or yes, no, etc.

    The Logistic regression is based on probability. In this case, the probability lies between 0 and 1. We use the log of the value to create a step function. The step function can distribute the values among the two classes.

    E.g. Let say we want to predict whether it will be a rainy day or not based on the temperature on a day. We have to give one of the two answers.

    True for a rainy day and false for a sunny day. We can use the temperature in our model to predict the probability of rain.

    Then we take the log of this probability to divide into two classes. It can be 0 or 1. We can use 0 for a sunny day and 1 for a rainy day.
    We can implement Machine Learning (ML) in such a way that it is independent of a specific model or learning algorithm.

    The model in this approach is a general model and is called algorithm independent.

    We can create a model that provides similar results even after using different types of algorithms.

    This helps in validating our approach and provides independent analyses approach for our model.
    “Use as filter” property in a chart, makes the data element in that chart act as a filter to other sheets/charts.

    On the desired chart, click on the pulldown menu on the right and select Use as Filter.

    A developer has created a dashboard using 4 sheets, say, FilledMap_color, Combo chart, Parameters and TableCalculation_Rank. How can he make sure that when the user clicks on a data element on one chart, the other charts are also filtered with that data element?

    Tableau Server licensing comes in two modes – User-based or core-based.

    User-based licensing restricts how many users can work on your installation of Tableau Server. Tableau Server can be installed on a single machine or multiple machines in a cluster.

    Core-based licensing provides a license on the number of cores on a single machine installation or multi-node installation. It does not restrict the number of users in the system.
    Tableau Data Extract (TDE) file is an extract of data and it depends on the size of the source data. As it is a compressed file, it will be smaller than the original size of the source data.
    A multi-developer environment is required when a number of dashboards are to be developed by different developers.

    In such an implementation/s, the team can be divided into data modelers and dashboard designers. Alternatively, one complete dashboard can be assigned to a single developer. But when the dashboard is huge then multiple developers can work on the same dashboard and create a visualization in separate workbooks.

    There can be many ways to implement such scenarios
    - Data modeler/s create data sources and save it as TDS file or publish it to the server. This will ensure that all data standards and transformations are maintained.
    - Developers connect to this published data source to develop dashboards.
    - If sheets are developed by different developers, then they can be merged into a single dashboard by performing copy/paste.
    - Once all sheets are merged into one workbook, there will be many data sources as data sources are tied to the sheets. These data sources are to be replaced by the primary data source and all other data sources can be closed.
    You can copy a visualization or a sheet from one Tableau workbook to another. The Sheet is tied to the data source, so when you copy the sheet, to another workbook, the data source will be copied as well. You can always replace this new data source with the one in the copied workbook and close the data source which is no longer in use.
    A Tableau application should be tested at multiple levels. The basic principle is to always compare against the source data

    - Check it against the original data source. Check the count of rows in Tableau by using the “Number of Records” measure and source table.

    - Compare the aggregations in the charts against the database SQL.

    - For excel data sources, use excel filters and formulas.

    - Export the data out of Tableau, by right-clicking on the chart and selecting view data

    How to test a Tableau application?

    Some of the ways to improve the performance of a Tableau dashboard

    - Use a performance recorder to gain knowledge of the performance of different sheets and actions in a workbook.

    - Use extracts, if possible. Extracts are faster than live connections.

    - Minimize the number of fields in your workbook. Hide fields that are not required.

    - Reduce the number of filters used on a worksheet and dashboard. In Tableau, filters are applied separately and each time a query is sent to the data source. Multiple filters will slow down the application.

    - All complex calculations should be performed at the data source level.

    - The number of sheets will affect performance. Delete or hide any unused sheets.
    The developer can right-click on the field and select Replace references and select the new field.
    The story uses dashboards and sheets to provide detailed information about a measure or KPI.
    Sheets or Dashboards make up a Story. A Story can contain multiple tabs. The story is created like any other sheet by clicking on the icon at the bottom of the screen
    What is Story in Tableau?

    Exercise: If you have created different sheets and dashboards using the previous exercises, create a Story by using FilledMap_Color, Db_SalesDashboard, and Sets sheets/Dashboards.
    Tableau workbook can be published to the Tableau Server by using Tableau Desktop.

    - Open your workbook on Tableau desktop.

    - -Navigate to Menu/Server and select Publish Workbook

    How will you deploy Tableau workbook on the server?



    - It will ask you for the server credentials. On successful login, you will get the options to publish your entire workbook or specific sheets.

    - Entire workbook or specific sheets in a workbook can be published.
    The above requirement can be fulfilled by creating a User Filter in Tableau Desktop during design.

    User Filter will work when Users login to the Tableau Server. To implement User Filter, you should have access to Tableau Server.
    To create a User filter, on Tableau Desktop, navigate to Menu-Server. Login to the Server.

    Navigate to Menu – Server – Create User Filter using the Segment.

    In this way when the User login’s he will see the data only for his segment

    In visualization, you have data for all the Segments. In the source data, Users are assigned to specific Segments. The requirement is that the users should see data only for their assigned Segments. What technique should be used to achieve this requirement?

     
    - Create an extract on Tableau desktop and configure it for Full refresh or incremental load.

    - Navigate to your sheet and right-click on your data source and select “Publish to Server”

    - On Publish Data Source to Tableau Server dialogue, you can set the schedule.

    How will you schedule a data extract in Tableau?

     

    - This will publish and set the schedule on the Server.

    - This schedule can be modified at any time.
    You have installed Tableau desktop personal edition. Personal edition allows you to connect to only excel and text files. All the other data sources are available with the licensed version.
    We should implement some practices that will ensure the enterprise-wide implementation

    - Data sources should be created only once. All the common transformations and calculations should be present in this data source.

    - The data source should be shared with other users creating the dashboards. Data Source/s can be shared by publishing them on the server or by creating a TDS file.

    - The look and feel of the dashboard should be the same. Color scheme, fonts, etc. should be decided and implemented in all the dashboards.

    - Create a dashboard review board that will review all the dashboards before moving to production. This will ensure that standards are maintained.
    Create an extract and choose the incremental load option.
    Tableau automatically aggregates measures on the view canvas. Default aggregation in Sum.

    When you drop a measure field, the aggregation applied is Sum.

    This aggregation can be changed to Average or any other type. When an “aggregated measure” is used in a calculation and then the calculation is placed on the view canvas, it shows Agg in front of the calculated column. It means that aggregation is built within the calculation. This aggregation cannot be changed.
    TDSX file will be a larger file as it is packaged with data.
    What-if scenario involves accepting user input in the dashboard and changing the calculations based on the user input. A good example will be discount offered, where the user wants to know, at run time, how his overall profitability will change if he changes the discount price.
    What-if analysis can be performed by the use of Parameters.
    On a scatter plot, highlight all such data elements and create a Group. This group will be displayed as a field in the Dimension section, which can be renamed.
    The size of the workbook depends on the specific requirement. There is no single size. of the dashboard depends on the data, the number of objects, and the functionality of the dashboard.
    If the dashboard size is larger than the RAM size of your computer then you can open it on the server which will have more RAM.
    - Performance recording feature records performance information on the various events on a dashboard as users interact with the worksheet and visualization.

    - This feature can be turned-on on the Tableau desktop and can be turned off once the recording is done.

    - This feature records performance information on the various events on a dashboard as users interact with the worksheets and visualization.

    - It creates a workbook under \Documents\My Tableau Repository\Logs. The name of this workbook is similar to performance_20160808T084552_11184.tab.

    - This workbook can be analyzed for performance metrics.

    - To turn-on performance recording, navigate to help on the menu and select.

    What is Performance Recording on Tableau desktop?

    This scenario can be implemented by creating a regular calculation. For example,

    In your visualization, you won't show - if the Sales is greater than 15000 then “Good Sales” else “Not so good sales”. What kind of calculation you will use to implement this rule.

    No. Green and Red in the above calculation do not mean color. It is just a calculated field/measure. After placing this calculation on the Color shelf, the automatic color will be displayed. To change to specific colors, the developer should navigate to Color shelf, click on Edit Colors, and set the desired colors.
    When you install Tableau desktop, it creates My Tableau Repository folder in your documents folder. The easiest way to back-up your previous work is to rename the existing “My Tableau Repository” folder and then uninstall or re-install Tableau
    Follow the steps below to understand a previously built workbook

    - Review the data sources. Edit the data source to see if any filter is used while extracting the data. Check to see if the connection is a live connection or an extract.
    - The developer may have renamed the dimensions and measures. Right-click on the dimensions and measures and select Describe getting information on the source columns.
    - Fields prefixed with = are calculated fields. Right-click and edit to understand their definition.
    - A delta symbol next to the measure pill in the view shows that Table Calculation is implemented. Edit Table Calculation to understand the definition.
    - At the bottom of the data, the window looks for Parameters and Sets. You can edit and view the definitions of these elements.
    - Right-click on the status bar and select “Unhide” to display the hidden sheets.
    - See the symbols of the data source to understand if they are extracts and if using data blending.
    - Check the output of the dashboards to see different sheets. See what data elements, objects, sheets are displayed on the dashboard.
    - Check for any aliases in the fields.
    - Look for the Actions in the dashboard. On the Menu, click on the “Dashboard” and navigate to “Actions”.
    - Locate the log files under “My Repository folder”.
    While creating a dashboard, care should be taken so that the dashboard is clean and easy to understand. The following practices can be followed

    - Design a dashboard for a specific Screen resolution. Discuss with your user community and decide on a screen resolution that will cater to all the user's

    - Design a Screen layout which makes the navigation on the screen natural and easy.

    - Avoid using complex calculations in the dashboard. Move such calculations to the data source.

    - Use muted colors. So that focus is on the dashboard not on the color.

    - Create a template and include all the layout related attributes on this template. Use this template to develop dashboards.

    The dashboard displays Quick filter, say, Category. By default, all the filters show (All) option. How can you turn this option off?



    From the pulldown menu of the quick filter, select Customize and uncheck “Show “All” Value.

    The dashboard displays Quick filter, say, Category. By default, all the filters show (All) option. How can you turn this option off?

    Tableau development should follow the Agile methodology.

    Traditional methodology or waterfall model, worked in sequential order – Feasibility study, Requirements, Design, Testing, Deployment, and support. This method though good for complex systems, considered to be time-consuming in answering business questions at a rapid pace.

    Agile methodology on the other hand facilitated the rapid development of the application. The development team worked closely with the business users and helped in faster delivery of the application.
    We use training and test datasets in the Supervised machine learning approach.

    The purpose of the training dataset is to discover the predictive relationship by using a model. With the training dataset, the model can learn the behavior of data and tweak itself. The model is built based on the data it discovers in the training dataset.

    Once the model is mature, we use the test dataset to get the accuracy of the hypothesis. Since the model has not seen the test dataset during the training phase, it gives us impartial results. Therefore, it is very important to keep the test dataset separate from the training data set.

    Sometimes the model gets the problem of overfitting the data. It can occur when training and test datasets get mixed up.
    Artificial Intelligence (AI) means machines can carry complex tasks in an intelligent way. It is an approach of making machines equipped with capabilities to execute complicated tasks that are generally done by expert humans. Machine learning (ML) is one of the applications of AI. In ML, we can provide data and training to machines to perform tasks in an intelligent way. Based on the data given to machines, they can train themselves for a specific task. In AI, we can have applied intelligence based on an algorithm. E.g. trading engines etc use AI to trade in high speed with high intelligence. In general, Artificially Intelligent systems can learn to do tasks by themselves. In ML, we design and develop algorithms to solve problems based on data already collected. In AI, we also cover other topics like- Natural Language Processing (NLP), Knowledge Representation (KR), Robotics, etc. In short, AI uses Machine Learning for solving complex problems.
    data +science