Knowledge Discovery and Data Mining I - Winter Semester 2018/19¶
- Lecturer: Prof. Dr. Thomas Seidl
- Assistants: Max Berrendorf, Julian Busch
Tutorial 0: A Short Introduction to Python¶
In this tutorial, we want to give you a short introduction to Python and some insights in the basic usage of some common libraries in the scope of Data Science. The tutorial is intended to prepare you for programming assignments on upcoming exercise sheets. There will be no live coding session in the tutorials, but there will be time for you to ask and discuss questions. The book "Dive Into Python 3" by Mark Pilgrim (http://www.diveintopython3.net/) is a great resource for self-study. If you need any help, feel free to contact your tutor or the assistants.
Installing Python¶
Anaconda distribution (recommended)
Visit the website https://www.continuum.io/downloads and download the Anaconda distribution for the Python 3.6 version. Make yourself familiar with the jupyter Python notebook which is included in the anaconda installation. If preferred, install also an IDE/editor of your choice, e.g. PyCharm. The Anaconda distribution comes along with a lot of libraries (numpy, scipy, pandas, ...) which otherwise have to be installed individually. If you need an additional library which is not initially included in Anaconda, you can install the lib via
conda install PACKAGENAME.Further information can be found in the documentation: https://conda.io/docs/index.html
Without distribution
Visit https://www.python.org/downloads/ and download your preferred python version. Next, go to https://pip.pypa.io/en/stable/installing/ and install $pip$. With $pip$ being installed one can install each package individually. For Example, to install the packages $numpy, scipy, matplotlib, ipython, jupyter, pandas$, execute the following command:
pip install --user numpy scipy matplotlib ipython jupyter pandasIf you need an additional package during the course, don't forget to install the libraries which might be handy.
Basic Python¶
Assigning Values to Variables. Create variables and assign numbers, strings, floating values to them.
prof = "Thomas Seidl"
no_studs = 13
temp = 13.0
print(prof)
Variable Types Python has five standard data types −
- Numbers
- String
- List
- Tuple
- Dictionary
Lists. Create a list which contains all numbers from 0 to 10
l0 = [1,2,3,4,5,6,7,8,9,10]
l0 = list(range(0,10))
l0
Loops and Conditionals. Using the created list, print each element of the created list if its is an odd number, by using a loop and conditionals. Try using different type of loops.
#This is a comment
'''
This is a block comment
'''
l1 = [x for x in range(10)]
print(l1)
# Version with for-loop
for e in l1:
if e%2 != 0:
print(e)
# Version w/ while-loop
i = 0
while i <len(l1):
# if l1[i] %2 != 0:
# print(l1[i])
if l1[i] & 1:
print (l1[i])
i+=1
#Version with list comprehension
l2 = [x for x in l1 if x%2 !=0]
print(l2)
List Comprehensions. Now, generate a list which contains all numbers from 0 to $n$ which have been squared using list comprehensions.
l3 = [x for x in [x**2 for x in range(7)] if x%2 !=0]
print(l3)
Functions. Write a function which takes an integer $n$. The function first creates a list of numbers from $0$ to $n$. Then, squares each number of the list. Further each of the squared numbers is tested if it is odd. All odd numbers are then appended to a new list. The function returns the list of odd (squared) numbers.
def get_odd(n):
return [x for x in [x**2 for x in range(n)] if x%2 !=0]
print(get_odd(7))
Assignments. Given a list $a=['I','like','cookies']$ and another list $b=a$. Replace in the list $b$ the word $'cookies'$ with $'apples'$. Finally, print both lists ($a$ and $b$). What do you observe? What leads to the observed behavior?
a = ['I','like','cookies']
b = a
b[2] = 'apples'
print("list a: "+str(a))
print("list b: "+str(b))
print(id(a),id(b))
Shallow Copy I. Given a list $a=['I','like','cookies']$ and another list which takes a shallow copy of $a$, $b=a[:]$. Like in the previous assignment, replace in the list $b$ the word $'cookies'$ with $'apples'$. Finally, print both lists ($a$ and $b$). What do you observe now?
a3 = ['I','like','cookies']
b3 = a3[:]
b3[2] = 'apples'
print("list a3: "+str(a3))
print("list b3: "+str(b3))
print(id(a3),id(b3))
print(id(a3[2]),id(b3[2]))
Shallow Copy II. Now, we are given a list $a = ['I', 'like', ['chocolate', 'cookies']]$. Another list $b = deepcopy(a)$ takes this time a deep copy from $a$. Change now the work $'cookies'$ with $'apples'$ in $b$. Print both lists ($a$ and $b$). What do you observe now?
Hint: For deep copy. first type: from copy import deepcopy
from copy import deepcopy
a4 = ['I','like',['chocolate', 'cookies']]
b4 = deepcopy(a4)
b4[2][1] = 'apples'
print("list a4: "+str(a4))
print("list b4: "+str(b4))
print(id(a4[2]),id(b4[2]))
Dictionaries I. Create a dictionary with $n$ entries, where the keys are enumerated from $0$ to $n-1$ and the values are their corresponding keys squared. Use list comprehensions.
Example for expected result: $n = 7; \{0:0, 1:1, 2:4, 3:9, 4:16, 5:25, 6:36\}$
d1 = {x : x**2 for x in range(7)}
print(d1)
Dictionaries II. Use the dictionary from the previous assignment. Write a list comprehension to get a list of all the keys of the dictionary.
#it actually corresponds to d.keys()
dlis = [d1[x] for x in d1]
print(dlis)
Lambda Functions. Write a list comprehension which takes a number $n$ and returns a list with even numbers, using a lambda function.
even1 = lambda x: x%2 ==0
l7 = [x for x in range(7) if even1(x)]
print(l7)
Map. First, write a function which takes a length in $inch$ and returns a length in $cm$. Given a list $l$ with lengths in $inches$: $l=[4,4.5,5,5.5,6,7]$. Write a list comprehension which takes $l$ and returns a list with all values converted to $cm$ using $map()$.
linch = [4,4.5,5,5.5,6,7]
def inch_to_cm(length):
return length*2.54
lcm = list(map(inch_to_cm, linch))
print(lcm)
Filter. Write a list comprehension which filters the list $l$ from the assignment above by returning only sizes between $4$ and $6$ $inches$.
lrange = list(filter(lambda x: x > 4 and x < 6, linch))
print(lrange)
Reduce. Write a list comprehension which reduces the list $l$ by summing up all lenghts.
Hint: for using the reduce function, you need to import it first by: from functools import reduce
from functools import reduce
lsum = reduce(lambda x,y: x+y, linch)
print(lsum)
List Reverse. Given the following list $a=[0,1,2,3,4,5]$. Write a function which reverses the list.
a = [0,1,2,3,4,5]
a[::-1]
Zipping of Lists. Given the following two lists, wher eone list represents the $x-Coordinate$ and another one the $y-Coordinate$:
- $xcoors = [0,1,2,3,4,5]$
- $ycoors = [6,7,8,9,10,11]$
Write a function which zips the two lists to a list of coordinate-tuples:
- $xycoors = [(0,6),(1,7),(2,8),(3,9),(4,10),(5,11)]$
xcoors = [0,1,2,3,4,5]
ycoors = [6,7,8,9,10,11]
zcoors = [99, 98, 97, 96, 95, 94]
#'manual zipping'
def manualzip(lisa, lisb):
reslis = []
for i in range(min(len(lisa),len(lisb))):
reslis.append((lisa[i],lisb[i]))
return reslis
print(manualzip(xcoors,ycoors))
print(list(zip(xcoors,ycoors, zcoors)))
Unzipping of Lists. Now, we are given a list of data points where the first dimension of each data point represents the age of a person and the second dimension the amount of money spent for chocolate per month in euro:
- $chocage = [(20,8), (33,18), (27,14),(66,23),(90,100)]$
Write a function which takes the list and separates it into two lists, one containing the ages and another one containing its corresponding amount of money spent for chocolate. The result would be e.g.:
- $age = [20,33,27,66,90]$
- $money\_spent = [8,18,14,23,100]$
chocage = [(20,8), (33,18), (27,14), (66,23), (90,100)]
#'manual unzipping'
def manualunzip(tuplelis):
lisa = []
lisb = []
for e in tuplelis:
a, b = e
lisa.append(a)
lisb.append(b)
return [tuple(lisa),tuple(lisb)]
print(manualunzip(chocage))
print(list(zip(*chocage)))
Object-oriented Programming¶
Object-oriented Programming I. We deal now with object-oriented programming in Python. For this purpose perform the following steps:
- Write a $Point$ class. A $Point$ class takes and $x$ and $y$ coordinate as an argument.
- Further this class shall have a setter method $setXY$ which takes and $x$ and $y$ coordinate and sets the attributes to the new provided values.
- The class shall also have a getter method $getXY$ which returns the current $x$ and $y$ coordiantes of the point.
- Write a method distance which takes another $point$ object and returns the euclidean distance between the provided point and the point itself. Hint: Take import math to use math.sqrt(value) in order to compute the square root.
import math
class Point(object):
def __init__(self, x, y):
#java: this.x = x;
self.x = x
self.y = y
def setXY(self, x, y):
self.x = x
self.y = y
def getXY(self):
return (self.x,self.y)
def distance(self, otherpoint):
d = (self.x-otherpoint.x)**2 + (self.y-otherpoint.y)**2
return math.sqrt(d)
Object-oriented Programming II. In a next step, the task is to create a class $Shape$. For this purpose perform the following steps:
- Create a class $Shape$ which takes a name and a color as parameters.
- Define a method $area$ which just returns $0.0$.
- Define a method $perimeter$ which just return $0.0$.
Now, create a class Rectangle which inherits from $Shape$ and in which you $implement$ the $area$ and $perimeter$ methods.
class Shape(object):
def __init__(self, name, color):
self.name = name
self.color = color
def area(self):
return 0.0
def perimeter(self):
return 0.0
class Rectangle(Shape):
def __init__(self, corner, width, height, color):
#super(...) 'equivalent':
Shape.__init__(self, "rectangle", color)
self.corner = corner
self.width = width
self.height = height
def perimeter(self):
return self.width*2 + self.height*2
def area(self):
return self.width * self.height
r = Rectangle(Point(4,4),10,5,'pink')
print('Perimeter of rectangle r: ',r.perimeter())
print('Area of rectangle r: ', r.area())
Numpy¶
Numpy I - Some Basic Functions. In this block, you will become familiar with the numpy library and some of its basic functionality. Please also consider to consult the documentation https://docs.scipy.org/doc/numpy-dev/index.html if needed. Solve the following tasks:
- Create an numpy array of floats containing the numbers from $0$ to $4$.
- Create the following matrix as a numpy matrix: $M = [[1,2,3], [4,5,6]]$.
- Get the shape of the matrix $M$.
- Check if the value $2$ is in $M$.
- Given the array $a = np.array([0,1,2,3,4,5,6,7,8,9], float)$. Reshape it to an $5\times2$ matrix.
- Transpose the previously introduced matrix $M$.
- Flatten matrix $M$.
- Given the array $b = np.array ([0,1,2,3], float)$. Increase the dimensionality of $b$.
- Create and $3\times3$ identity matrix.
import numpy as np
#create an np array with float as type
arr0 = np.array([1,2,3,4], float)
arr0
#create a 2x3 matrix using np arrays
arr1 = np.array([[1,2,3],[4,5,6]], float)
arr1[0,0]
#get shape of an array
arr1.shape
#getting type of array
arr1.dtype
#check if a particular value is in the array
[1,2,3] in arr1
#reshape an array e.g. 1x10 to an 5x2 array
arr2 = np.array(range(10), float)
#print(arr2)
arr3 = arr2.reshape((5,2))
#print(arr3)
#fill matrix with specific value
arr4 = np.array(range(10))
arr4.fill(42)
print(arr4)
#transpose an array
arr5 = np.array([[1,2,3],[4,5,6]], float)
arr6 = arr5.transpose()
print(arr5)
print(arr6)
#flatten an array...
print(arr6.flatten())
#increasing dimensionality of an array
arr7 = np.array([1,2,3],float)
print(arr7)
print(arr7[:,np.newaxis])
#array of ones and zeros
print("array of ones and zeros")
print(np.ones((2,3),float))
print(np.zeros((2,3),float))
#getting an identity matrix
print(np.identity(3,float))
Numpy II - Linear Algebra and Statistics. This assignemtn has its focus on numpy function of the linear algebra and statistics domain. Solve the following tasks using numpy:
- Given the following two numpy array: $a=np.array([1,2,3], float)$, $b=([4,5,6],float)$. Compute the dot product of $a$ and $b$
- Given the following matrix $M = [[1,2,3], [4,5,6], [7,8,9]]$, compute the determinant of $M$ by using the $linalg$ package of the numpy library.
- Compute the eigenvalues and eigenvectors of $M$
- Compute the inverse of $M$
- Given the numpy array $c=np.array([1,4,3,8,3,2,3], float)$, compute the mean of $c$
- using $c$, compute the median.
- given the following matrix $C=[[1,1], [3,4]]$, compute the covariance of $C$.
# DOT PRODUCT
arr8 = np.array([1,2,3],float)
arr9 = np.array([4,5,6],float)
print(np.dot(arr8,arr9))
# DETERMINANT
arr10 = np.array([[1,2,3],[4,5,6],[7,8,9]],float)
print(np.linalg.det(arr10))
# COMPUTE EIGENVALUES AND EIGENVECTORS
eigenvals, eigenvecs = np.linalg.eig(arr10)
print(eigenvals)
print("------")
print(eigenvecs)
# COMPUTE INVERSE
print(np.linalg.inv(arr10))
# COMPUTE MEAN AND MEDIAN
arr11 = np.array([1,4,3,8,9,2,3],float)
print("mean: ",np.mean(arr11))
print("median: ",np.median(arr11))
# COMPUTE COVARIANCE
arr12 = np.array([[1,1],[3,4]],float)
print('cov: ',np.cov(arr12))
Matplotlib¶
Matplotlib - Plotting Figures in Python. In this assignment we are finally going to become familiar with the plotting library of Python. For this we solve the following tasks below. Please consider to consult the documentation if needed: https://matplotlib.org/contents.html.
- Given a list of data points : $dpts=[(3,3),(4,5),(4.5,6),(9,7)]$. Plot the function using $plt.plot(xcoors, ycoors)$
- You are given two tiny clusters $c_1 = [(1,2),(3,1),(0,1),(2,2)]$ and $c_2=[(12,9),(8,10),(11,11), (14,13)]$. Plot them in a scatter plot using $plt.scatter(xcoors, ycoors)$, where $c_1$ and $c_2$ have different colors. The $x-axis$ represents the time spent at a parking lot in hours, and the $y-axis$ represents the money spent in euro. Create axis labels for your figure.
- Take the two clusters $c_1$ and $c_2$ together and compute their pairwise distances, storing them in a matrix. Plot the resulting matrix as a heatmap using $plt.imshow(my\_matrix, cmap='coolwarm')$.
import matplotlib.pyplot as plt
%matplotlib inline
#1
dpts = np.asarray([(3,3),(4,5),(4.5,6),(9,7)])
#access second column (y-coordinates)
print(dpts[:,1])
plt.figure()
plt.plot(dpts[:,0],dpts[:,1])
plt.ylabel('y-axis')
plt.xlabel('x-axis')
plt.show()
#2 scatter plot
c1 = np.array([(1,2),(3,1),(0,1),(2,2)])
c2 = np.array([(12,9),(8,10),(11,11),(14,13)])
plt.figure()
plt.scatter(c1[:,0],c1[:,1], color='r')
plt.scatter(c2[:,0],c2[:,1], color='b')
plt.xlabel('time spent at parking lot [h]')
plt.ylabel('money spent [€]')
plt.title("Fancy studies")
plt.show()
#3 now for something completely different: heatmap...
from scipy.spatial import distance
distmx = []
for e in c1:
newrow = []
for f in c2:
d = distance.euclidean(e,f)
newrow.append(d)
distmx.append(newrow)
for e in distmx:
print(e)
plt.imshow(distmx, cmap='coolwarm', interpolation='nearest')
plt.colorbar()
plt.show()
Pandas¶
Pandas- Basic Data Analysis. For this assignment, we will use the file moviemetadata.csv, which contains entries from the IMDB movie database. The original source of the data is Kaggle: https://www.kaggle.com/deepmatrix/imdb-5000-movie-dataset/. Please also consider to consult the documentation http://pandas.pydata.org/pandas-docs/stable/ if needed. Solve the following tasks:
- Read the csv file as a DataFrame for further processing using $pandas.read csv()$.
- Inspect the read csv file using $.shape$, $.columns$, $.info()$ and $.describe()$.
- Display the first five records of the data set using $.head(5)$ and the last five records using $.tail(5)$.
- Select from the data set the first five records. Those records shall only contain the following columns: $movie\_title$, $duration$ and $num\_voted\_users$.
- Select the first five movies containing the genre $Action$. Display only the columns $movie\_title$ and $genres$.
- Sort the action movies by their $imdb\_score$ and display the names and scores the top-10 scored movies.
- Group the movies by column $director$ and display the top-10 directors with the highest mean gross of their movies.
- Optional: Delete all rows, which contain at least one missing value. Visualize parts of the data using $pandas.plotting.scatter\_matrix$ and $groupby.DataFrameGroupBy.hist$.
import pandas as pd
from pandas.plotting import scatter_matrix
import matplotlib.pyplot as plt
plt.style.use('default')
%matplotlib inline
# Read movie dataset
movie_data = pd.read_csv('moviemetadata.csv',
delimiter=',',
header=0,
decimal='.')
# Get an overview
print('== OVERVIEW ==')
print(movie_data.shape)
print(movie_data.columns)
movie_data.info()
display(movie_data.describe())
# Show first/last 5 records
display(movie_data.head(5))
display(movie_data.tail(5))
# Indexing
print('== INDEXING ==')
display(movie_data[['movie_title', 'duration', 'num_voted_users']].head(5))
# Filtering
print('== FILTERING ==')
action_mask = movie_data['genres'].str.contains('Action')
display(movie_data[action_mask][['movie_title', 'genres']].head(5))
# Sorting
print('== SORTING ==')
display(movie_data[action_mask].sort_values('imdb_score', ascending=False)[['movie_title', 'imdb_score']].head(10))
# Grouping
print('== GROUPING ==')
display(movie_data.groupby(['director_name'])['gross'].mean().sort_values(ascending=False).head(10))
# Delete rows with NaNs
print('== DELETION OF NAN ROWS ==')
print(movie_data.shape)
movie_data = movie_data.dropna(axis=0, how='any')
print(movie_data.shape)
# Visualization
print('== VISUALIZATION ==')
scatter_matrix(movie_data[['director_facebook_likes', 'budget', 'gross', 'imdb_score']], alpha=0.2, figsize=(10, 10), diagonal='kde')
movie_data.groupby('color')['title_year'].hist(alpha=0.4, figsize=(10, 10))
