MIT 6006 introduction to algorithm video note

思丝健康特产，干果、坚果、花草茶等天然健康产品，欢迎您的光临！

亚马逊店：http://t.cn/RyN9r0m
京东店：http://mall.jd.com/index-31856.html
当当店：http://shop.dangdang.com/12661
一号店：http://t.cn/RyNCtdV
淘宝店：https://ssjktc.taobao.com

19:29 2013-12-3 Tuesday

start introduction to algorithm, lec 1
algorithmic thinking, peak finding
19:30 2013-12-3
///////////////////////////////////////////////
9:44 2013-12-4 Wednesday
start introduction to algorithm, lec 1
algorithmic thinking, peak finding
9:45 2013-12-4
peak finding, both 1D & 2D!
9:46 2013-12-4
computational complexity
9:46 2013-12-4
asymptotic complexity
9:46 2013-12-4
asymptotically less complex
9:47 2013-12-4
US highway system
9:47 2013-12-4
large input
9:49 2013-12-4
cubic complexity
9:50 2013-12-4
we concern about efficient procedures for
solving large scale problems!
9:51 2013-12-4
scalability
9:51 2013-12-4
classic data structures
9:51 2013-12-4
BST == Binary Search Tree
9:52 2013-12-4
balanced binary search tree
9:52 2013-12-4
but they stood the test of time
9:53 2013-12-4
classical algorithms
9:54 2013-12-4
real implementation in phthon!
9:54 2013-12-4
challenging & worthwhile
9:55 2013-12-4
algorithmic thinking
9:55 2013-12-4
peak finding
9:57 2013-12-4
event simulation
9:57 2013-12-4
Genome comparison
9:59 2013-12-4
numerics: RSA encryption
9:59 2013-12-4
https, SSL
10:03 2013-12-4
graphs: Rubik's cube
10:05 2013-12-4
shortest path
10:05 2013-12-4
DP == Dynamic Programming:
image compression
10:07 2013-12-4
peak finder
10:13 2013-12-4
What is a peak?
position 2 is a peak iff b >= a && b >= c
10:14 2013-12-4
one-dimensional peak finder
10:16 2013-12-4
straightfoward algorithm
10:22 2013-12-4
I search long & hard, I search exhaustively
10:23 2013-12-4
worst-case complexity
10:25 2013-12-4
O(n) is just the upperbound
10:26 2013-12-4
asymptotic complexity of this problem is linear!
10:26 2013-12-4
one-dimensional peak finder
10:27 2013-12-4
problem: find a peak if it exists
10:30 2013-12-4
so what we have here is a recursive algorithm
10:34 2013-12-4
you want to write an argument that this
algorithm is right!
10:35 2013-12-4
formal argument
10:37 2013-12-4
recurrence relation
10:37 2013-12-4
divide & conquer algorithm
10:40 2013-12-4
log to the base of 2 n
10:42 2013-12-4
huge difference
10:42 2013-12-4
reduce complexity
10:43 2013-12-4
two dimensional array peak finder
10:45 2013-12-4
What is a 2D-peak?
a is a 2D-peak iff
a >=b, a >= d, a >= c, a >= e
10:46 2013-12-4
greedy ascent algorithm
10:50 2013-12-4
you have to make a choice where to start
10:52 2013-12-4
you also make a choice of what the default
directions are?
11:00 2013-12-4
greedy ascent algorithm
11:04 2013-12-4
touching a bunch of elements
11:05 2013-12-4
starting point + direction of search
11:08 2013-12-4
worst-case complexity
11:09 2013-12-4
try to generalize the binary search algorithm
into the 2D version
11:18 2013-12-4
it's efficient & incorrect
11:20 2013-12-4
2D peak may not exist
11:26 2013-12-4
greedy ascent algorithm doesn't work for 2D!
11:27 2013-12-4
global maximum
11:31 2013-12-4
solve the new problem with half of the columns!
-------------------------------------------------------
12:53 2013-12-4
start introduction to algorithm, lec 2
models of computation
13:02 2013-12-4
What is an algorithm?
computational procedure for solving a problem
input -> algorithm -> output
15:29 2013-12-4
style of programming
15:30 2013-12-4
Random Access Machine(RAM)
15:34 2013-12-4
load / store
15:35 2013-12-4
cache
15:38 2013-12-4
What is a pointer machine?
we have dynamically allocated objects // heap
15:46 2013-12-4
random access machine
pointer machine
15:48 2013-12-4
OOP == Object-Oriented Machine
15:47 2013-12-4
What is python's model?
1. "list" == array
2. OOP
15:50 2013-12-4
table doubling
16:00 2013-12-4
time to compare
16:00 2013-12-4
dict: key, value pair
16:04 2013-12-4
long: long integer
16:07 2013-12-4
document distance
16:08 2013-12-4
document distance problem
16:08 2013-12-4
d(D1, D2)
16:09 2013-12-4
are these document roughly identical?
16:09 2013-12-4
use automated test for cheating
16:09 2013-12-4
simalarity between documents on internet
16:10 2013-12-4
thing a document as a sequence of words
16:11 2013-12-4
idea: use shared words
16:12 2013-12-4
think of document as vector
D[W] == number of occurence of w in D
16:18 2013-12-4
if you have a high dot product, you have a lot
in common!
16:20 2013-12-4
the angle between the two vectors
16:26 2013-12-4
iterate throught the words in the document
16:27 2013-12-4
dictionary runs in O(1) with high probability
16:29 2013-12-4
exponential time
-----------------------------------------------
16:32 2013-12-4
introduction to algorithm, lec 3
insertion sort & merge sort
16:32 2013-12-4
merge sort which is a divide & conquer algorithm
16:36 2013-12-4
why sorting?
lots of obvious example
16:38 2013-12-4
problems become easy once it's sorted:
find a median
16:41 2013-12-4
as long as you have the comparison function
16:43 2013-12-4
it's constant time if you have a sorted list
16:43 2013-12-4
binary search
16:44 2013-12-4
linear search -> logarithmic search
16:47 2013-12-4
data compression
16:48 2013-12-4
rendering
16:52 2013-12-4
insertion
16:53 2013-12-4
pairwise swaps
16:54 2013-12-4
by pairwise swaps down to the right position
17:05 2013-12-4
compare & swap
17:07 2013-12-4
comparison >> swaps comparison cost!
17:13 2013-12-4
constant improvement
17:14 2013-12-4
merge sort
17:16 2013-12-4
recursive algorithm
17:17 2013-12-4
the merge routine
17:19 2013-12-4
Merge: two sorted array as input
17:24 2013-12-4
two finger algorithm for merge
17:29 2013-12-4
recursion tree
17:32 2013-12-4
recursion tree expansion
17:35 2013-12-4
What levels in this tree are? and leaves?
17:40 2013-12-4
insertion sort is in-place sorting
17:42 2013-12-4
auxilliary space
17:44 2013-12-4
in-place merge sort
17:47 2013-12-4
asymptotic complexity
-------------------------------------------------
17:54 2013-12-4
introduction to algorithm, lec 4
heaps & heap sort
17:55 2013-12-4
priority queue
20:03 2013-12-4
ADT == Abstract Data Type
20:03 2013-12-4
so today's ADT is a heap
20:07 2013-12-4
What is a heap?
a heap is an implementation of a priority queue
20:07 2013-12-4
heap is an array visualized as a nearly complete binary tree
20:08 2013-12-4
binary tree
20:09 2013-12-4
nearly complete binary tree
20:09 2013-12-4
full complete binary tree
20:12 2013-12-4
heap representation of an array
20:13 2013-12-4
Heap as a tree?
root of tree first element (i == 1)
parent(i) == i / 2
left(i) == 2 * i, right(i) == 2 * i + 1
20:15 2013-12-4
Max-Heap property
20:15 2013-12-4
What is a Max-Heap property?
the key of a node is >=
the keys of its children
20:20 2013-12-4
we want to maintain the Max-Heap property
when we trying to modify the heap!
// invariant
20:21 2013-12-4
extract_max() // take the max element & delete it!
20:22 2013-12-4
the big question is: how do we maintain the
Max-Heap property as we modifying the heap?
20:23 2013-12-4
it's not a max-heap, not a min-heap, it's
just a heap!
20:26 2013-12-4
Heap operations?
1. build_max_heap:
produced a heap from an unordered array
20:28 2013-12-4
max_heapify:
correct a single violation of the heap property
in a subtree's root
20:29 2013-12-4
What it does is takes something that is not a max_heap,
and fix that!
20:31 2013-12-4
What is Max_heapify?
Assume that the trees rooted at left(i) & right(i) are
max heaps // precondition
20:37 2013-12-4
MAX_HEAPIFY(A, 2)
20:38 2013-12-4
exchange with bigger child
20:39 2013-12-4
you realize that there is a violation of the max_heap
property at this node(root)!
20:41 2013-12-4
so what is the complexity of max_heapify()?
20:45 2013-12-4
nearly complete binary tree
20:47 2013-12-4
max_heapify() is O(log(n)) in terms of complexity
20:47 2013-12-4
heap-based sorting algorithm
20:47 2013-12-4
convert an array into a max-heap?
the keyword here is max-heap, because every
array can be visualized as a heap
20:51 2013-12-4
observation: A[n/2+1...n] are all leaves!
20:56 2013-12-4
can you do a better analysis?
20:58 2013-12-4
increasing operations as you get higher & higher
up, but there is less & less nodes!
21:00 2013-12-4
observe: max_heapify() takes O(1) for nodes that
are 1 level above leaves and in general O(l) times
for nodes that are l levels above the leaves
21:02 2013-12-4
observe 2: n/4 nodes with level 1, n/8 with level 2,
1 node with lgn level
21:04 2013-12-4
sum up all these work across all these levels
21:09 2013-12-4
it's a convergent series that are bounded by a
constant!
21:09 2013-12-4
that's the key observation
21:12 2013-12-4
the argument is made that this is a convergent
series & bounded by a constant!
21:15 2013-12-4
build max-heap from unordered array
21:16 2013-12-4
MHP == Max-Heap Property
21:17 2013-12-4
we can run max-heapify to fix this
-----------------------------------------
21:24 2013-12-4
introduction to algorithm, lec 5,
BST, BST sort
21:24 2013-12-4
BST == Binary Search Tree
21:28 2013-12-4
Runway reservation system:
Airport with a single runway
21:30 2013-12-4
so reservation for future landings is
really what this system is built for
21:40 2013-12-4
unsorted list/array
21:44 2013-12-4
insertion point of the sorted array
21:51 2013-12-4
max-heap or min-heap are actually have a fairly
weak invariant!
21:52 2013-12-4
max-heap or min-heap are essentially
partial ordered trees!
22:01 2013-12-4
fast insertion into sorted array
22:04 2013-12-4
Invariant for BST: for all nodes x, if y is in the left
subtree, then key(y) <= key(x)
if y is in the right subtree, you have
key(y) >= key(x)
22:21 2013-12-4
multiset: 2 elements have the same key!
22:24 2013-12-4
find_min() // keep going to the left
find_max() // keep going to the right
O(h) complexity, h is the height of the tree
22:25 2013-12-4
augmented binary search tree
22:25 2013-12-4
augmentation
22:28 2013-12-4
that'the notion of augmentation
22:29 2013-12-4
add a little bit of more information to
the node structure
22:29 2013-12-4
augment the BST structure
22:39 2013-12-4
subtree sizes
22:52 2013-12-4
balanced binary search tree
//////////////////////////////////////////////////////////////////
9:15 2013-12-5 Thursday
start introduction to algorithm, lec 06
AVL tree, AVL tree sort
9:15 2013-12-5
inorder traversal
9:20 2013-12-5
O(h) // h == height of the BST
9:20 2013-12-5
balanced binary tree
9:27 2013-12-5
What is a balanced tree?
we call a tree is balanced if h == O(lgn)
9:28 2013-12-5
AVL tree
9:29 2013-12-5
height of a node
9:29 2013-12-5
What is the height of a node?
the height of a node is the length
of longest path from it down to a leaf
9:34 2013-12-5
formula for the height of a node:
height of a node ==
max{height(left child), height(right child)} + 1
9:34 2013-12-5
it's obvious useful for computing
9:34 2013-12-5
data structure argmentation
9:35 2013-12-5
DS == Data Structure
9:42 2013-12-5
What is an AVL tree?
require heights of left & right children
of every node to differ by at most +-1
9:44 2013-12-5
height of the subtree == height of the root node
9:48 2013-12-5
AVL trees are balanced // means h == O(lgn) 85671829 gangtie
10:35 2013-12-5
Nh == minimuum number of nodes in an AVL tree of height h
Nh == 1 + N(h-1) +N(h-2)
10:37 2013-12-5
this looks like Fibonacci recurrence!
10:46 2013-12-5
AVL insert
1. we do the simple BST insertion
2. fix AVL property
10:53 2013-12-5
rotations!
10:58 2013-12-5
left rotate, right rotate
11:14 2013-12-5
How to fix AVL property?
from the changed node up, suppose x is lowest
node violating AVL
11:17 2013-12-5
assume right(x) is higher
if x's right child is right-heavy or balanced
11:27 2013-12-5
AVL sort:
insert n items
in-order traversal
11:28 2013-12-5
ADT == Abstract Data Type
11:30 2013-12-5
balanced BST
--------------------------------------------------
11:35 2013-12-5
review AVL tree, AVL sort
BST property
11:57 2013-12-5
height of tree
height of node
12:09 2013-12-5
height of the subtree(height of its node)
12:09 2013-12-5
subtree
12:12 2013-12-5
AVL tree are balanced!
12:20 2013-12-5
If I fix the height to h, what is the minimum
nodes?
12:23 2013-12-5
all I want is a recursive formula
12:27 2013-12-5
it's almost Fibonacci
12:40 2013-12-5
AVL insert
12:42 2013-12-5
How to fix AVL property?
12:43 2013-12-5
the different between left & right size?
12:47 2013-12-5
let me tell you rotations, super cool!
12:48 2013-12-5
left_rotate(x)
12:51 2013-12-5
in a constant number of pointer changes, I
can go from this to that!
12:51 2013-12-5
still have BST property
12:55 2013-12-5
our beauty is restored!
12:56 2013-12-5
this is more annoying // the zig-zag path
12:57 2013-12-5
How can I fix this zig-zag?
double rotate: left rotate, right rotate
13:01 2013-12-5
this is some people called dobule rotation
13:01 2013-12-5
BST property, AVL property
13:08 2013-12-5
left-heavy, right-heavy, even
13:15 2013-12-5
AVL sort: using in-order traversal
13:16 2013-12-5
ADT == Abstract Data Type
13:19 2013-12-5
there are many priority queues:
heap, AVL
13:19 2013-12-5
balanced BST
13:19 2013-12-5
the main reason you prefer heap is that
it's inplcae, don't use any extra memory!
----------------------------------------------
13:21 2013-12-5
start introduction to algorithm, lec 7
Couting sort, Radix sort, Lower bounds
for sorting & searching
13:21 2013-12-5
theory, proof, counta-example
13:38 2013-12-5
model of computation
13:39 2013-12-5
a new kind of "model of computation":
comparison model
13:40 2013-12-5
What is a comparison model?
* all input items are black boxes(ADTs)
* only operations allowed are comparisons
* time cost == #comparisons
13:44 2013-12-5
so to prove that we're going to introduce the
notion of "decision tree"
13:44 2013-12-5
decision tree
13:46 2013-12-5
What is a decision tree?
Decision Tree:
any comparison algorithm can be viewed as a tree
of all possible outcomes & theire outcomes & the
resulting answer
13:47 2013-12-5
binary tree
13:54 2013-12-5
internal node represents a binary decision
// comparison
13:54 2013-12-5
decision tree <-> algorithm
13:54 2013-12-5
a leaf in the tree corresponds to "found answer"!
13:57 2013-12-5
downward root-to-leaf path
13:59 2013-12-5
What is the worst-case running time of the decision tree?
the height of the tree(root)
14:00 2013-12-5
running time
14:00 2013-12-5
How low can you made your tree?
14:02 2013-12-5
searching lower bounds:
n preprocessed items, finding a given item,
among them in comparison model requires Ω(lgn)
in worst case
14:09 2013-12-5
proof:
decision tree is binary, it must have n leaves,
one for each answer
14:11 2013-12-5
if you has a binary tree with n leaves,
the height of tree is at least log(n)
14:12 2013-12-5
binary tree is optimal
14:12 2013-12-5
BST is a good way to solve problems!
14:19 2013-12-5
internal node
14:24 2013-12-5
How do you know that height >= log(n!)?
log(n * (n-1) * (n-2) ... * 1) ==
log(n) + log(n-1) + log(n-2) + ... + log(1)
14:32 2013-12-5
Taylor's series approximation
14:35 2013-12-5
once you set up this framwork of comparison
trees, become a question of height of comparison tree!
14:40 2013-12-5
RAM == Random Access Machine
14:40 2013-12-5
comparison model
14:41 2013-12-5
Linear-Time sorting // integer sorting
14:43 2013-12-5
How does Linear-Time sorting works?
assume n keys soring are integers ∈{0, 1, ..., k-1}
(&each fits in a word)
14:47 2013-12-5
can do a lot more than comparisons
for k, .... can sort in O(n) times // linear time sorting!
14:48 2013-12-5
when k is not enormous, can sort easily with
linear time!
14:49 2013-12-5
Counting sort
14:49 2013-12-5
What is a Counting Sort?
count all items,
just traverse the array of counters
15:06 2013-12-5
radix sort
15:06 2013-12-5
a much cooler example is called radix sort!
it's going to use counting sort as a subroutine
and it's going to get a much larger part of key
and still get a linear time complexity!
15:07 2013-12-5
How does radix sort works?
* imagine each integer as base b
* if we know the largest integer is k, then
#digits == d == logb(k)
* then sort integers by the least significant digits(LSB)
.
.
.
* sort by MSB
15:21 2013-12-5
sort by digit using counting sort
15:25 2013-12-5
what base b should be?
---------------------------------------------
start introduction to algorithm, lec 8
Hashing with chaining
15:29 2013-12-5
Dictionary: Abstract Data Type(ADT)
15:48 2013-12-5
search, insert, delete
15:51 2013-12-5
there are 2 kinds of databases:
Hashing, search tree
15:53 2013-12-5
search tree
15:54 2013-12-5
compiler & interpreter
15:57 2013-12-5
in the network of a machine
15:57 2013-12-5
substring search
16:01 2013-12-5
Direct-Access table
16:11 2013-12-5
gigantic memory hog
16:12 2013-12-5
memory hog
16:14 2013-12-5
prehashing
16:14 2013-12-5
this is a notion called prehashing
16:14 2013-12-5
What is a prehashing function?
prehashing maps a key into nonnegative integers
16:16 2013-12-5
in theory keys are finite & discrete
16:19 2013-12-5
in Python, hash(x) is the prehash of x
16:37 2013-12-5
keyspace
16:40 2013-12-5
pigeonhole principle
16:40 2013-12-5
this is what we called collision
16:42 2013-12-5
there are two methods dealing with collisions:
chaining & open addressing
16:45 2013-12-5
What is a chaining?
chaining is linked list of colliding elements
16:49 2013-12-5
the reason is randomization
16:51 2013-12-5
simple uniform hashing
16:52 2013-12-5
space of keys // key space
16:54 2013-12-5
What is a simple uniform hashing?
each key is equally likely to be hashed
to any state of the table, independent
of where other keys hashing
17:00 2013-12-5
What is a load factor?
expected length of chain for n keys, m slots
== n/m == alpha == load factor
17:04 2013-12-5
then you walk the linked-list...
17:05 2013-12-5
you get constant running-time for all your operations
17:18 2013-12-5
universal hashing
-------------------------------------------------------
//////////////////////////////////////////////////////
10:30 2013-12-6 Friday
start introduction to algorithm, lec 9
table doubling, Karp-Rabin
10:31 2013-12-6
to grow the table
10:58 2013-12-6
amortization
10:58 2013-12-6
What is a amortization?
11:01 2013-12-6
if m > n: grow table
11:01 2013-12-6
grow / shrink
11:01 2013-12-6
What is gow table?
allocate the memory & rehash
------------------------------------------------
download 6.004 computation structure(2 version), 6.02
introduction to digital communication videos
at the internet bar!
///////////////////////////////////////////////////
17:52 2013-12-7 Saturday
start introduction to algorithm, lec 9
table doubling, Karp-Rabin
17:55 2013-12-7
hashing with chaining
17:57 2013-12-7
simple uniform hashing, universal hashing
18:00 2013-12-7
load factor
18:00 2013-12-7
grow the table
18:03 2013-12-7
idea: start small, grow / shrink as necessary
18:03 2013-12-7
amortization
18:04 2013-12-7
if m > n: grow table
18:06 2013-12-7
allocate the memory & rehash
18:15 2013-12-7
table doubling
18:19 2013-12-7
this is an idea we called amortization
18:20 2013-12-7
What is an amortization?
* operation takes "T(n) amortized" if k operations
takes <= k * T(n) time
18:21 2013-12-7
the expected running time is T(n)
18:21 2013-12-7
the amortized running time is T(n)
18:28 2013-12-7
still get constant amortized
18:29 2013-12-7
that's on the right track
18:34 2013-12-7
amortized running time
18:35 2013-12-7
resizable array
18:36 2013-12-7
python list is a resizable array
18:39 2013-12-7
string matching
18:40 2013-12-7
you want to search for some substring in
a giant text string
18:57 2013-12-7
What we're looking at is a rolling window of t!
19:00 2013-12-7
Rolling hash ADT
19:04 2013-12-7
string matching with Rolling hash ADT
19:04 2013-12-7
Karp-Robin algorithm
----------------------------------------------------
19:38 2013-12-7
start introduction to algorithm, lec 10
open addressing, cryptographic hashing
19:38 2013-12-7
the simplest way of implementing a hash table:
open addressing
19:42 2013-12-7
hash table with chaining <-> open addressing
19:44 2013-12-7
cryptographic hashing
19:45 2013-12-7
chaining hashtables, open addressing hashtables
19:45 2013-12-7
cryptographic hashing is very different from
regular hashing
19:46 2013-12-7
What is an open addressing?
this another approach to dealing with collisions
open addressing has no chaining
19:49 2013-12-7
m >= n // m the number of slots, n the number of elements
19:50 2013-12-7
probing
19:50 2013-12-7
What is a probing?
we need a Hash function specifies order of
slots to probe for a key(for insert/search/delete)
19:53 2013-12-7
universe of keys, trial count
19:57 2013-12-7
k is arbitrary key
h(k, 1), h(k, 2), ..., h(k, m-1) to be a permutation of
0, 1, ..., m-1
20:04 2013-12-7
oops! this slot is occupied
20:05 2013-12-7
the first probe actually failed!
20:05 2013-12-7
h(496, 1) == 4 failed!
h(496, 2)?
20:11 2013-12-7
keep probing until an empty slot is found,
insert item when found!
20:13 2013-12-7
How to search key in open addressing hashing?
Search(k): As long as the slots encountered are occupied
by keys != k, keep probing until you either encounter k
or found empty slot
20:17 2013-12-7
once in a while, we want to delete a key!
20:23 2013-12-7
fail incorrectly
20:25 2013-12-7
solution: Replace deleted item with DeleteMe
flag(different from None)
20:32 2013-12-7
insert treats DeleteMe the same as None,
but search keep going!
20:36 2013-12-7
so let's talk about probing strategies!
* Linear probing
20:37 2013-12-7
What is a Linear probing?
h(k, i) == (h'(k) + i) mod m
// h'(k) is an ordinary hash function
20:38 2013-12-7
Does that satisfy the permutation property?
20:41 2013-12-7
What is cluster?
consecutive groups of occupied slots which keeps longer
20:44 2013-12-7
you just increase your cluster length by 1
20:46 2013-12-7
double hashing
20:46 2013-12-7
you have to run a couple of hashing
20:46 2013-12-7
What is a Double Hashing?
h(k, i) == (h1(k) + i * h2(k)) mod m
// h1() & h2() are just ordinary hash functions
20:50 2013-12-7
if h2(k) is relatively prime to m => permutation
so we choose m == exp(2, r), h2(k) for all k is odd
20:51 2013-12-7
Uniform Hashing Assumption // NOT simple uniform hashing!
20:52 2013-12-7
What is a Uniform Hashing Assumption?
Each key is equally likely to have any one of
the m! permutations as its probe sequence
20:53 2013-12-7
this is very hard to put into practice!
20:55 2013-12-7
as alpha(the load factor) increases, the expected
number of probes becomes large!
20:56 2013-12-7
chaining table
20:57 2013-12-7
open addressing is easy to implement, and use
less memory, use no pointer!
20:59 2013-12-7
because these DeleteMe flags, they still hurt you!
21:03 2013-12-7
FYI == For Your Interest
21:03 2013-12-7
cryptographic hashing
21:04 2013-12-7
so maybe the best way to motivating this is
through an example!
21:04 2013-12-7
password storage
21:06 2013-12-7
one-way:
given hash value h(x) == Q, it's very hard to find x!
----------------------------------------------------------
21:15 2013-12-7
start review introduction to algorithm, lec 10
open addressing
21:15 2013-12-7
implement hashtable using a single array data structure!
this is open addressing!
21:17 2013-12-7
hashtable chaining
21:19 2013-12-7
chaining hashtables, open addressing hashtables
21:20 2013-12-7
open addressing another approach to dealing with
collisions!
21:22 2013-12-7
at least one item per slot
no chaining
21:23 2013-12-7
m >= n // m: number of slots, n: number of elements
21:23 2013-12-7
the notion of "probing"
21:23 2013-12-7
h(k, i) // k == key, i == trial count
21:28 2013-12-7
for arbitrary key k, h(k, 1), h(k, 2), ..., h(k, m-1)
to be a permutation of 0, 1, 2, ... , m-1
21:32 2013-12-7
out hash function will tell us what that sequence is,
so we don't need any pointers etc!
21:38 2013-12-7
oops! this slot is occupied
21:39 2013-12-7
the first probe actually failed!
-----------------------------------------------------------
21:45 2013-12-7
start introduction to algorithm, lec 11
irrational
21:45 2013-12-7
high-precision computation
21:46 2013-12-7
64 bits word length
21:46 2013-12-7
this is a short module on numerics
21:47 2013-12-7
irrational numbers to arbitrary precision
21:47 2013-12-7
RSA encryption
21:49 2013-12-7
irrational number: which can not be represented as p/q
21:51 2013-12-7
so they try to find patterns in irrationals
21:52 2013-12-7
so that's another motivation for high-precision arithmetic
21:55 2013-12-7
Catalan numbers
21:56 2013-12-7
What is a Catalan number?
Set P of balanced parentheses strings
21:58 2013-12-7
the nice thing about this definition is that
you can get every nonempty balanced parentheses strings
via Rule 2 from a unique alpha, beta pair
22:01 2013-12-7
the cardinality of the set gets bigger & bigger
22:01 2013-12-7
balanced parentheses strings
22:13 2013-12-7
Newton's method
22:14 2013-12-7
successively approximation
22:25 2013-12-7
quadratic convergence
22:26 2013-12-7
correct numbers increases by 2 with each iteration
22:30 2013-12-7
How to compute a/xi?
actually first takes the reciprocol of xi, and
multiply by a!
22:31 2013-12-7
most division algorithm use the multiplication
as a subroutine!
22:33 2013-12-7
digits of precision
22:36 2013-12-7
high-precision multiplication
----------------------------------------------------------
//////////////////////////////////////////////////////
10:16 2013-12-8
start review introduction to algorithm, lec 10
10:19 2013-12-8
open addressing
------------------------------
10:29 2013-12-8
start review lec 11,
Integer Arithmetic, KaraTsuba Multiplication
10:35 2013-12-8
word length
10:35 2013-12-8
high-precision computation
10:42 2013-12-8
Catalan numbers
10:43 2013-12-8
Catalan numbers has a recursive definition
10:43 2013-12-8
it's the cardinality of a set
10:44 2013-12-8
What is the set P?
Set P of balanced parenthese strings
1.λ∈P (λ is the empty string)
2. If α,β∈P, then (α)β∈P
11:21 2013-12-8
quadratic convergence, linear convergence
11:22 2013-12-8
quadratic rate of convergence
11:28 2013-12-8
high-precision multiplication
11:29 2013-12-8
digits of precision
11:33 2013-12-8
going from division to multiplication
11:34 2013-12-8
high-precision multiplication
11:37 2013-12-8
x1 is the more significant half,
x0 is the low half
11:38 2013-12-8
recursive multiply
---------------------------------------------
11:41 2013-12-8
start introduction to algorithm, lec 12
square roots, Newton's method
11:41 2013-12-8
quadratic rate of convergence
12:32 2013-12-8
geometric progression
12:39 2013-12-8
Error analysis of Newton's method
12:42 2013-12-8
logarithmic number of iterations
12:43 2013-12-8
multiplication algorithm
* naive divide & conquer
12:45 2013-12-8
at each level of the recursive tree
12:45 2013-12-8
you can play a few mathematical tricks
12:46 2013-12-8
recurrence relationship
12:48 2013-12-8
Toom-Cook algorithm
12:49 2013-12-8
break a number up to 3 parts
12:51 2013-12-8
Toom-3
12:55 2013-12-8
FFT == Fast Fourier Transform
Schonhege-Stranen
13:04 2013-12-8
High-precision Division
13:22 2013-12-8
quadratic convergence
13:25 2013-12-8
ask a question if you're confused
13:32 2013-12-8
here is the key observation
13:40 2013-12-8
complexity of computing square roots
------------------------------------------
13:47 2013-12-8
start introduction to algorithm, lec 13
BFS == Breadth First Search
13:48 2013-12-8
Graph search: explore a graph
13:53 2013-12-8
Rubik's cube
13:53 2013-12-8
initial state -> target state, you want to find the paths?
13:55 2013-12-8
What is a graph?
graph G == (V, E)
V == Vertices, E == Edges
13:56 2013-12-8
directed graph, undirected graph
14:00 2013-12-8
Application of graph:
* web crawling
* social networking
* network broadcasting
* garbage collection
* model checking
* checking mathematical conjecture
14:07 2013-12-8
pocket cube: 2 * 2 * 2
14:07 2013-12-8
configuration graph // or configuration space
14:19 2013-12-8
has vertex for each possible state of the cube
14:20 2013-12-8
#vertices == 8! * exp(3, 8)
14:20 2013-12-8
8! because you can essentially permute these
cubes in your.., and each of them has 3 different
twists
14:22 2013-12-8
that is vertices, let's talk about edges!
14:23 2013-12-8
every move takes you from one configuration to another
14:24 2013-12-8
all of the configurations, you can reach by 1 move
14:25 2013-12-8
these are all states you can reach by two moves
14:26 2013-12-8
the diameter of the graph
14:35 2013-12-8
graph representation:
adjacency list, adjacency matrix
14:39 2013-12-8
incoming edges, outgoing edges
14:40 2013-12-8
graph exploration problems
14:40 2013-12-8
implementation of adjacency list
14:42 2013-12-8
you can do it in a more object-oriented fashion:
v.neighbours() == adj[v]
14:50 2013-12-8
BFS == Breadth-First Search
the goal is to visit all nodes reachable
from given starting point
14:53 2013-12-8
we want to achieve O(V + E) time
14:54 2013-12-8
until we went out of graph
14:55 2013-12-8
careful to avoid duplicates
15:04 2013-12-8
so that's how you avoid duplicates
15:04 2013-12-8
first frontier, 2nd frontier
15:07 2013-12-8
so this becomes the frontier of level 3
15:10 2013-12-8
parent pointers
15:11 2013-12-8
these pointers always form a tree
15:11 2013-12-8
shortest path
15:15 2013-12-8
the parent pointer actually give you the
shortest path, that's the nice part of BFS!
15:16 2013-12-8
eventually you'll get to the solved state
15:16 2013-12-8
optimal strategy
15:16 2013-12-8
What is the running time of this algorithm?
15:18 2013-12-8
Handshaking Lemma
----------------------------------------------
19:14 2013-12-8
start introduction to algorithm, lec 14,
DFS(Depth-First Search), Topological Sort
19:14 2013-12-8
BFS is layer by layer
19:19 2013-12-8
what is Depth-First Search?
recursively explore graph, backtracking as necessary
19:30 2013-12-8
parent pointer
19:33 2013-12-8
DFS, DFS_visit
19:49 2013-12-8
edge classification
* tree edge: parent pointer
visit new vertex via edge
* forward edge
* backward edge
* cross edges
19:54 2013-12-8
What is a forward edge?
a forward edge goes from a node to a descendent
in a tree
19:55 2013-12-8
What is a backward edge?
backward edge goes from a node to ancestor in a tree
19:56 2013-12-8
What is a cross edge?
a cross edge goes between two different siblings
20:01 2013-12-8
detecting forware edges is a little trickier
20:03 2013-12-8
Which edges exists in undirected graph?
20:08 2013-12-8
cycle detection
20:08 2013-12-8
topological sort
20:09 2013-12-8
cycle detectin: does my graph has cycles?
G has a cycle <=> DFS has a back edge
20:29 2013-12-8
Job Scheduling:
given DAG(Directed Acyclic Graph), order vertices
so that all edges point from lower order to higher order
20:33 2013-12-8
What is a Topological Sort?
run DFS, output the reverse of finishing times
of vertices
20:36 2013-12-8
we take the reverse of the finishing order
---------------------------------------------------
20:40 2013-12-8
start introduction to algorithm, lec 15,
single-source shortest path problem
20:41 2013-12-8
optimal substructure
20:45 2013-12-8
asymptotic complexity
20:45 2013-12-8
shortest path problem
20:46 2013-12-8
G(V, E, W)
// G == Graph, V == Vertex, E == Edge, W == Weight
20:47 2013-12-8
Dijkstra algorithm
20:48 2013-12-8
talking about a simple graph, what is the
asymptotic relationship between V & E? // Vertices & Edge
20:49 2013-12-8
complete graph
20:49 2013-12-8
What is a complete graph?
a complete graph is a graph with edges between
any pair of vertices
20:50 2013-12-8
simple graph <-> multi-graph
20:51 2013-12-8
Bellmen-Ford algorithm
20:53 2013-12-8
negative weights
20:55 2013-12-8
weight of the path is simply the summation
of weight of the edges
20:55 2013-12-8
dynamic range of the weights
21:01 2013-12-8
you have an exponential number of the paths
21:02 2013-12-8
optimal substructure property
21:03 2013-12-8
weighted graph
21:03 2013-12-8
the shortest path weight from u to v as delta(u, v)
21:08 2013-12-8
iterative process
21:17 2013-12-8
predecessor relationship
21:18 2013-12-8
minimum-weight path
21:21 2013-12-8
d(v) // value inside circle, current weight
Π[v] // predecessor on best path to v, Π[s] == NIL
21:22 2013-12-8
predecessor relationship & current distance
21:25 2013-12-8
negative weights
21:25 2013-12-8
What is the motivation for a graph with a negative weights?
* reverse toll
* social network
21:34 2013-12-8
negative weight cycle
21:42 2013-12-8
so that's your termination condition, it's
different from the termination condition of
the nonnegative edge graph
21:42 2013-12-8
detect cycles
21:47 2013-12-8
relaxation of the edge(u, v)
21:51 2013-12-8
I can relax this edge, and update 6 to 5!
and also update the predecessor relationship
21:53 2013-12-8
faily clean termination condition:
you cannot relax your edge any more!
21:55 2013-12-8
until all edges have d[v] <= d[u] + w(u, v)
// edges cannot be relaxed any more!
21:56 2013-12-8
relaxation condition
22:05 2013-12-8
relax this edge // edge relaxation
22:09 2013-12-8
optimal substructure
22:09 2013-12-8
subpaths of a shortest path are shortest paths
------------------------------------------------
22:13 2013-12-8
start introduction to algorithm, lec 16
Dijkstra algorithm
22:14 2013-12-8
relaxation
22:19 2013-12-8
shortest path in DAG(Directed Acyclic Graph)
22:20 2013-12-8
S.P. == Shortest Path
22:21 2013-12-8
d[v]: length of the current S.P. from source S to V
22:21 2013-12-8
through the process we called relaxation, we continually
reduce this d values
22:22 2013-12-8
δ(s, v): length of a shortest path
22:23 2013-12-8
Π[v]: is a predecessor of v in the shortest path from s to v
22:29 2013-12-8
basic relaxation operation:
Relax(u, v, w)
if d[v] > d[u] + w(u, v)
d[v] = d[u] + w(u, v) // update current value
Π[v] = u // update predecessor relationship
22:31 2013-12-8
I want to show that relaxation is safe
22:33 2013-12-8
Lemma: the relaxation operation maintains the invariant
that d[v] >= δ(s, v) for all vertices
22:38 2013-12-8
triangle inequality
22:42 2013-12-8
the triangle inequality has to do with the shortest path,
not the single edge weights
22:48 2013-12-8
w(u, v) which is a direct way, single edge way
22:54 2013-12-8
DAGs == Directed Acyclic Graph
22:54 2013-12-8
it's not the negative edges that cause the problem,
it's the negative cycles!
22:55 2013-12-8
it's only you get into the negative cycle iteration
that cause the indeterminate state!
22:56 2013-12-8
* Topological sort the DAG path from u to v, implies
that u is before v in the ordering
* one pass over vertices in topological sorted order
relaxed each edge that leaves each vertex
23:08 2013-12-8
DP == Dynamic Programming
23:09 2013-12-8
so we got one algorithm under our belt!
23:16 2013-12-8
greedy algorithm
23:29 2013-12-8
priority queue
23:31 2013-12-8
priority are the d() values
23:41 2013-12-8
extract min operation
23:45 2013-12-8
binary min-heap
23:47 2013-12-8
Fibonacci heap
23:47 2013-12-8
amortized time
-----------------------------------------------------
////////////////////////////////////////////////////////
10:34 2013-12-9 Monday
review introduction to algorithm, lec 14
DFS, topological sort
10:35 2013-12-9
DFS == Depth-First Search
10:51 2013-12-9
recursively explore the graph, backtracking as necessary
10:56 2013-12-9
DFS is the top-level algorithm,
DFS_visit is the recursive part
10:56 2013-12-9
this will find all the clusters, all the
strongly connected component part of your graph
10:58 2013-12-9
edge classification
11:00 2013-12-9
let's talk about running time
11:11 2013-12-9
handshaking lemma
11:11 2013-12-9
BFS is good for shortest path
11:13 2013-12-9
edge classification:
* tree edge(parent pointer) // bold edges
// 3 other nonbold edges
* forward edges: node -> descendent
* backward edges: node -> ancestor
* cross edges
11:22 2013-12-9
every parent pointer denote a reverse of a tree edge
11:26 2013-12-9
which of the edges exist in undirected graphs?
tree edge, forware edge, backward edge, cross edge
11:27 2013-12-9
you can never have a forward edge in undirected graph!
11:33 2013-12-9
why do I care these edge classification?
cycle detection, topological sort
11:34 2013-12-9
What is a cycle detection?
G has a cycle <=> DFS has a back edge
11:35 2013-12-9
back edge == backward edge
11:37 2013-12-9
ancestor, descendent
11:38 2013-12-9
do you see a cycle? a directed cycle
11:46 2013-12-9
depth-first nature of DFS
11:50 2013-12-9
balanced parentheses
11:53 2013-12-9
recursion stack
11:54 2013-12-9
Does my graph has a cycle?
11:56 2013-12-9
Job scheduling // topological sort
given DAG(Directed Acyclic Graph), order vertices,
so that all edges point from lower order to higher order
11:57 2013-12-9
What is a DAG intuitively?
I got to put my socks on before put my shoes on
12:03 2013-12-9
topological sort: run DFS, output the reverse
of finishing times of vertices
12:03 2013-12-9
we use this top-level DFS, so everybody gets visited
12:06 2013-12-9
backward edges are annoying, all edges here are
forward edge, tree edge, cross edge!
12:06 2013-12-9
same kind of arguments
----------------------------------------------------
12:12 2013-12-9
review introduction to algorithm, lec 15
single-source shortest path problem
12:13 2013-12-9
weighted graphs
12:13 2013-12-9
negative edges
12:13 2013-12-9
optimal substructure
12:13 2013-12-9
if you allow weights on graph edges
12:17 2013-12-9
shortest path algorithm
12:19 2013-12-9
optimal substructure property
12:20 2013-12-9
G(V, E, W)
G == Graph, V == Vertex, E == Edge, W == Weight
12:24 2013-12-9
Dijkstra algorithm
12:26 2013-12-9
+ve == nonnegative
12:27 2013-12-9
nonnegative weighted edges
12:27 2013-12-9
in general, what is the asymptotic relationship
between V & E?
an upperbound: E == O(V*V)
12:31 2013-12-9
simple graph
12:31 2013-12-9
What is a simple graph?
at most one edge between any pair of vertices
12:31 2013-12-9
simple graph <-> multigraph
12:31 2013-12-9
What is the running time of Dijkstra's algorithm?
O(VlgV + E)
12:33 2013-12-9
Bellmen-Ford algorithm
it runs negative/positive edges, this has
a complexity O(VE)
12:33 2013-12-9
a path is a sequence of edges
12:41 2013-12-9
tries to find a path p that has minimum weight
12:41 2013-12-9
it's not the function of weight
12:45 2013-12-9
you have an exponential number of path
12:50 2013-12-9
the dynamic range of the weights are exponential!
12:50 2013-12-9
weighted graph
12:53 2013-12-9
What is δ(u, v)?
δ(u, v) is the shortest path weight from u to v
12:54 2013-12-9
my goal is find δ
12:56 2013-12-9
initially everything is infinity, because you haven't
found any path yet
12:57 2013-12-9
What is d(x)?
d() is the current weight of node x
13:03 2013-12-9
what I want to do is to decrease this d number
13:18 2013-12-9
my job is done, when my d number shrinks down to
the correct δ values
13:18 2013-12-9
try to iterative reduce these numbers
13:18 2013-12-9
discover the shortest path throught that process
13:22 2013-12-9
it gets more complicated when you have negative edge
13:22 2013-12-9
predecessor relationship
13:23 2013-12-9
What is d(v)?
d(v) is the value in the circle, which is the
current weight
13:24 2013-12-9
What is Π[v]?
Π[v] is the predecessor vertex on the best path to v
13:27 2013-12-9
current best path
13:27 2013-12-9
predecessor relationship & the current distance
13:28 2013-12-9
negative weights
13:29 2013-12-9
Why do these things exist?
* reverse toll
* social network
13:33 2013-12-9
is there a way to shifting these weights to
make it more positive?
13:35 2013-12-9
try to get rid of the negative edges as possible
as you can! because the asymptotic complexity
13:36 2013-12-9
negative weight cycles
13:39 2013-12-9
exponential number of path
13:39 2013-12-9
negative cycles
13:39 2013-12-9
eventually you want to terminate
13:39 2013-12-9
but what you even didn't terminate
13:40 2013-12-9
you get this annoying cycle here, that's an
negative weight cycle!
13:41 2013-12-9
but if you have a graph with negative cycle,
this is the recipe for an infinite-loop
13:42 2013-12-9
because the termination condition isn't set properly
13:43 2013-12-9
What would you want Bellmen-Ford to do here?
13:44 2013-12-9
everybody buy that?
13:44 2013-12-9
you don't touch a negative weight, but more importantly
you don't touch a negative cycle!
13:45 2013-12-9
the negative cycle makes the shortest path indeterminate!
13:54 2013-12-9
so that's your termination condition
13:54 2013-12-9
detect negative cycles
13:55 2013-12-9
notion of "relaxation"
13:55 2013-12-9
we talk about a single source here
13:56 2013-12-9
What is "relaxation"?
"Relax" edges(u, v) if d[v] > d[u] + w(u, v)
then you set d[v] = d[u] + w(u, v)
13:58 2013-12-9
I can update the value of d[v]
13:58 2013-12-9
you keep doing this(repeat) until all edges
have d[v] <= d[u] + w(u, v)
13:59 2013-12-9
the assumption here is that no negative cycles!
13:59 2013-12-9
because 5 < 6, I can relax this edge & update
the current value d!
14:03 2013-12-9
fundamental notion used in every algorithm
14:03 2013-12-9
What is the termination condition?
until all edges have d[v] <= d[u] + w(u, v)
// until all edges cannot be relaxed
14:04 2013-12-9
exponential number of paths could cause a problem!
14:10 2013-12-9
dynamic range of the weights
14:10 2013-12-9
the point here is that...
14:11 2013-12-9
relax this edge
14:15 2013-12-9
so how are we going to do better than that?
14:17 2013-12-9
this is the pathological ordering
14:18 2013-12-9
optimal substructure
14:18 2013-12-9
subpaths of shortest path are shortest path
14:18 2013-12-9
optimal substructure property
14:21 2013-12-9
-------------------------------------------------
14:23 2013-12-9
review introduction to algorithm, lec 16
Dijkstra algorithm
14:24 2013-12-9
shortest path in DAGs
14:30 2013-12-9
shortest path in graphs w/o negative edges
14:30 2013-12-9
S.P. == Shortest Path
14:32 2013-12-9
What is d[v]?
d[v] is the length of the current shortest path
from source s to v
14:32 2013-12-9
we continuously reduce these d values!
14:33 2013-12-9
What is δ values?
δ(u, v) is the length of a shortest path
14:34 2013-12-9
what is Π(v)?
Π(v) is the predecessor of v in the shortest path
from s to v
14:35 2013-12-9
then we have to worry about the asymptotic complexity
of the algorithm?
14:41 2013-12-9
two special cases:
one for DAGs, one for nonnegative edges
14:41 2013-12-9
How the relaxation process goes?
Relax(u, v, w):
if d[v] > d[u] + w(u,v)
d[v] = d[u] + w(u, v)
Π[v] = u
14:44 2013-12-9
update not only the current value, but also
the predecessor relationship
14:45 2013-12-9
I want to show that relaxation is safe!
14:45 2013-12-9
Lemma:
the relaxation operation maintains the invariant
that d[v] >= δ(s, v) for all v ∈V
14:48 2013-12-9
pick an edge, relax it....
14:49 2013-12-9
what this lemma says is that: you can never get something
in the middle that is less than shortest path value...
14:50 2013-12-9
we want to show that relaxation operation doesn't
makes d[v] incorrect!
14:52 2013-12-9
d[u] >= δ(s, u)
that's the induction hypothesis, now by the triangle
inequality δ(s, v) <= δ(s, u) + δ(u, v)
14:54 2013-12-9
What is δ(s, v)?
δ(s, v) is a path between s & v, it's the best path,
that's the "shortest path"
14:55 2013-12-9
What is w(u, v)?
w(u, v) is the direct way & single-edge way
of going from u to v!
15:03 2013-12-9
w(u, v) >= δ(u, v)
15:04 2013-12-9
we have a relaxation algorithm that is safe!
15:05 2013-12-9
apply some sequence of relaxations
15:05 2013-12-9
DAG == Directed Acyclic Graph
15:06 2013-12-9
DAG: we cannot have (negative)cycles
15:06 2013-12-9
it's not the negative edges that cause the problem,
it's the negative cycles!
15:07 2013-12-9
indeterminate shortest path value!
15:08 2013-12-9
topological sort the DAG, path from u to v implies
that u is before v in the ordering
15:08 2013-12-9
linear ordering
15:09 2013-12-9
one pass over vertices in topological sorted order
relaxing each edge that leaves each vertex
15:12 2013-12-9
shortest path in DAGs
15:12 2013-12-9
works for any choice of starting vertex
15:13 2013-12-9
topological sort a DAG
15:17 2013-12-9
starting vertex
15:19 2013-12-9
everything going to left is marked with infinity
15:19 2013-12-9
final value
15:24 2013-12-9
DP == Dynamic Programming
15:24 2013-12-9
the reason for that is that we don't have cycles!
15:24 2013-12-9
this algorithm works for DAGs!
15:25 2013-12-9
Dijkstra algorithm
15:26 2013-12-9
it's nontrivial to prove its correctness
15:26 2013-12-9
Dijkstra algorithm is a greedy algorithm
15:26 2013-12-9
iteratively build the shortest path
15:26 2013-12-9
greedy algorithm
15:32 2013-12-9
priority queue
15:35 2013-12-9
Dijkstra(G, W, s) // G == Graph, W == Weight, s == source
15:35 2013-12-9
S is the set of vertices we know the shortest path to it!
this is an invariant
15:43 2013-12-9
Q is the set of vertices that are to be processed!
15:43 2013-12-9
while(Q is not empty)
extract min from Q
15:45 2013-12-9
starting vertex
15:47 2013-12-9
What is Relax(u, v, w)?
when extract min, update current value & predecessor value
15:48 2013-12-9
Q is a priority queue, the priority are the d() values!
15:51 2013-12-9
initially d[s] == 0, all the other vertices's d()
are infinity!
15:51 2013-12-9
then you relax all other vertices outgoing from s!
15:55 2013-12-9
d[A] == 0, d of B ~ E are all infinity!
15:55 2013-12-9
once you select A from S, you relax all edges coming
from A!
16:02 2013-12-9
it's like doing an breadth search, but being greedy!
16:03 2013-12-9
because when I'm doing a relax here, I'm doing
a decrease-key operation!
16:09 2013-12-9
decreasing-key operations
16:10 2013-12-9
implementing the priority queue
16:10 2013-12-9
insert, extract_min, decrease_key
16:12 2013-12-9
What data structure should we use?
we can use a binary min-heap!
16:15 2013-12-9
this is not the optimal complexity of Dijkstra!
16:15 2013-12-9
Fibonacci heap
16:15 2013-12-9
amortized analysis
16:16 2013-12-9
two special cases:
* DAGs
* graph without negative edges(Dijkstra)
-------------------------------------------------
16:17 2013-12-9
start introduction to algorithm, lec 17
Bellman Ford
16:18 2013-12-9
the general S.P. algorithm, even with negative cycles!
16:31 2013-12-9
Bellman-Ford is polynomial time algorithm
16:31 2013-12-9
-ve == negative
16:32 2013-12-9
this happened to be negative cycle!
16:32 2013-12-9
all of these edges have positive weight
16:33 2013-12-9
to detect this negative cycle exists
16:37 2013-12-9
so let's take a 2nd look at our Generic Shortest
Path algorithm!
16:38 2013-12-9
initialize for v ∈V,
d[v] <-- ∞, // current value
Π[v] <--NIL // predecessor value
16:40 2013-12-9
Relax edge(u, v, w):
if d[v] > d[u] + w(u, v)
update d[v] // current value
update Π[v] // predecessor value
16:42 2013-12-9
relax the edge
16:42 2013-12-9
you just repeatedly
1. select edge
2. relax edge(u, v, w)
until you cannot relax anymore
16:43 2013-12-9
complexity could be exponential time,
even for positive edge!
16:45 2013-12-9
this may not even be terminated if there
is a negative weighted cycle reachable
from the source
16:46 2013-12-9
polynomial time is great, exponential time is bad!
16:48 2013-12-9
What is a Bellman-Ford(G, W, s)?
G == Graph, W == Weight, s == source
Bellman-Ford(G, W, s)
Initialize()
for i = 1 to |V| - 1
for each edge(u, v) ∈E
Relax(u, v, w)
// |v| is the number of vertices!
16:52 2013-12-9
Relax(u, v, w) // relax operation
17:00 2013-12-9
check:
foreach edge(u, v) ∈E
if d[v] > d[u] + w(u, v)
then report negative cycle exists
17:01 2013-12-9
if no negative cycles, Bellman-Ford will find the shortest path,
otherwise it will detect negative weight cycles if it exists!
17:06 2013-12-9
Theorem: If G == (V, E) contains no negative weight cycles,
then after Bellman-Ford finished execution,
d[v] = δ(s, v) for all v ∈V
17:09 2013-12-9
Corollary: if a value d[v] failed to converge after |v|-1 passed,
there exists a negative cycle reachable from s
17:15 2013-12-9
shortest simple path
17:15 2013-12-9
shortest simple path
-------------------------------------------------------------------
////////////////////////////////////////////////////////////
10:06 2013-12-10
review introduction to algorithm, lec 17
Bellman-Ford
10:06 2013-12-10
negative weights cycle
10:14 2013-12-10
Generic Shortest Path problem
repeated select edge, Relax edge(u, v, w)
until no edges can be relaxed
10:23 2013-12-10
shortcoming:
* complexity could be exponential time
* this algorithm may not be terminated
// if there is a negative cycle reachable from the source
10:24 2013-12-10
polynomial time is great, exponential time is bad!
10:29 2013-12-10
Bellman-ford can fix the 2nd problem
10:27 2013-12-10
Bellman-Ford(G, W, s)
Initialize()
for i = 1 to |v|-1 // for all vertices
foreach edge(u, v) ∈E
Relax(u, v, w)
10:30 2013-12-10
What is RelaxEdge()?
Relax(u, v, w)
if d[v] > d[u] + w(u, v)
d[v] = d[u] + w(u, v) // update current value
Π[v] = u // update predecessor value
10:32 2013-12-10
if no negative cycles in the graph, then you got
the shortest path
else, there is negative cycle
10:33 2013-12-10
foreach edge(u, v) ∈E
if d[v] > d[u] + w(u, v)
then report negative cylce exists
10:35 2013-12-10
Theorem: If G == (V, E) contains no negative weight cycles,
then after B-F executes d[v] = δ(s, v) for all v∈V
10:43 2013-12-10
Corollary: If a value d[v] fails to converge after |v|-1
passes, there exists a negative weight cycle reachable
from s
10:44 2013-12-10
we're looking for shortest simple path
10:48 2013-12-10
proof by induction
10:50 2013-12-10
Let v ∈V, p = <v0, v1, v2,..., vk> v0 == s to vk == v
10:52 2013-12-10
k <= |v|-1 else there exists negative cycles
10:52 2013-12-10
this path p is a shortest path with minimum number of edges
10:52 2013-12-10
there may be many shortest path, but I'm going to pick
the one with the minimum number of edges!
10:53 2013-12-10
no negative cyles => p is simple => k <= |v|-1
10:57 2013-12-10
so I'm just growing the frontier one node every pass!
10:57 2013-12-10
optimal substructure property
10:58 2013-12-10
subset of a shortest path is a shortest path
10:59 2013-12-10
after 1 pass throught E, we have d[v1] = δ(s, v1),
because we will relax edge(v0, v1)
11:03 2013-12-10
after 2 passes, d[v2] = δ(s, v2), because in 2nd pass,
we will relax (v1, v2)
11:04 2013-12-10
after |v|-1 passes, all reachable vertices have δ values!
11:04 2013-12-10
Bellman-Ford(G, W, s)
Initialize()
for i = 1 to |v|-1 // for all vertices
foreach edge(u, v) ∈E
Relax(u, v, w)
11:06 2013-12-10
s vertex // source
11:21 2013-12-10
depending on the order which you relax
11:22 2013-12-10
in the first pass, the ordering of the relaxation
is ....
11:23 2013-12-10
edges are ordered in a certain way, at each time
you're going to relax edges in the exact same order
11:24 2013-12-10
if you relax this edge, nothing happens!
11:24 2013-12-10
at the 2nd pass, you start with this vertex again,
because that's the ordering
11:25 2013-12-10
w is the dynamic range of the weights
11:28 2013-12-10
initialization is affected by s, because d[s] = 0,
while others are started infinity
11:31 2013-12-10
relaxation is not affected by s!
11:31 2013-12-10
shortest path <-> longest path
11:33 2013-12-10
shortest path, longest path have this nice duality
11:37 2013-12-10
simply by negating each of these edges, and run
Bellman-Ford
11:38 2013-12-10
I cannot compute shortest path because it's undefined!
11:41 2013-12-10
if negative weight cycles do not exist
we report the shortest simple path
11:43 2013-12-10
shortest simple path reaches v from s
11:43 2013-12-10
if you have a negative weight cycle in your graph,
it's a NP hard problem to find the shortest simple path!
11:46 2013-12-10
it just means we don't know any algorithm that is
better than exponential time!
11:47 2013-12-10
in the sense that they're both NP hard proble,
if you can solve one, you can solve the other!
11:48 2013-12-10
you're allowed to abort. you detect the problem that there is a
negative weight cycle
-------------------------------------------------------------------
12:41 2013-12-10
start introduction to algorithm, lec 18
speeding up Dijkstra
12:41 2013-12-10
improve real world performance
12:42 2013-12-10
you can optimize Dijkstra's performance
in the case of "one source, one target" situation
12:44 2013-12-10
bidirectional search
12:44 2013-12-10
bidirectional Dijkstra
12:44 2013-12-10
all pairs shortest path
12:45 2013-12-10
to do some coding optimization
12:50 2013-12-10
if you know what the specific target is, you simply
sotp if u == t!
12:50 2013-12-10
so obviously this will run faster!
12:51 2013-12-10
What is a bidirectional search?
Alternate forward search form s,
backward search from t(following edges backward)
12:58 2013-12-10
you go forward & backward...
12:58 2013-12-10
as you growing your backward frontier
12:59 2013-12-10
What is df[u], db[u]?
df[u]: distance for forward search
db[u]: distance for backward search
13:01 2013-12-10
priority queue:
Qf: forward search
Qb: backward search
13:01 2013-12-10
Q: What is the termination condition?
// terminate the search when the frontiers meet
some vertex u has been processed both in the forward search
& the backward search
deleted(extracted) from both Qf & Qb!
13:05 2013-12-10
// Πf, Πb, Πf[v1] = s
Q: How do we find the shortest path from s to t?
claim: If w was processed first from both Qf, Qb
find shortest path using Πf from s to w, then
find shortest path using Πb from t to w(backwards)
13:17 2013-12-10
predecessor
13:20 2013-12-10
continually apply Πf to w until we get to s,
continually apply Πb to w until we get to t
13:22 2013-12-10
have a vertex that cause the termination which
are called w
13:23 2013-12-10
using the forward pointers & the backward pointers
13:23 2013-12-10
but the problem is that:
w may not be on the shortest path!
13:24 2013-12-10
we have to do something that more than the
termination condition
13:24 2013-12-10
What is incorrect is that to use w to construct
the shortest path!
13:25 2013-12-10
these fontiers are just barely collided!
13:26 2013-12-10
alternate forward search & backward search
13:31 2013-12-10
so far so good
13:33 2013-12-10
asap == as soon as possible
13:33 2013-12-10
hashed horizontally means that it has been removed from Qf,
hashed vertically means that it has been removed from Qb
13:36 2013-12-10
people buy that?
13:45 2013-12-10
frontiers collide at the shortest length path
the shortest length path is not the shortest weight path!
13:46 2013-12-10
shortest weight path
13:46 2013-12-10
termination condition does not change
13:47 2013-12-10
find an x(possibly different from w) which has
minimum value of df[x] + db[x]
13:54 2013-12-10
which is the minimum sum of the forward & backward
priorities!
13:55 2013-12-10
Goal-Directed Search:
Modify edge weights with potential functions
*w(u, v) = w(u, v) - λ(u) + λ(v)
13:58 2013-12-10
but the basic idea with λ is that...
13:58 2013-12-10
source vertex
13:58 2013-12-10
single source, single target
14:00 2013-12-10
What nodes you want to make uphill,
what nodes you want to make downhill?
14:01 2013-12-10
any weight w(p) is going to modified according its
destination & source
14:02 2013-12-10
potential method
14:05 2013-12-10
what is used to be shortest path should stay
a shortest path
14:05 2013-12-10
so the nice thing is any path from s to t is
going to get shifted the same amount corresponding
to this additional value!
14:07 2013-12-10
one way of get these potential function is
to use a landmark
14:07 2013-12-10
the basic idea is you had landmark l ∈V, // vertex l
precompute λ(u, l)
14:13 2013-12-10
for any input vertex, you want to find the shortest
path to this landmark
14:14 2013-12-10
change the source, but the destination stays the same
--------------------------------------------------------
14:17 2013-12-10
start introduction to algorithm, lec 19
Dynamic Programming I: Fibonacci, Shortest Paths
14:18 2013-12-10
DP == Dynamic Programming
14:31 2013-12-10
it's very good for optimization problem
14:33 2013-12-10
you can think dynamic programming as a kind
of exhaustive search
14:33 2013-12-10
DP is approximately "careful brute force"
14:34 2013-12-10
memorization & subproblem
14:34 2013-12-10
How to compute Fibonacci numbers?
14:37 2013-12-10
DP == subproblems + "reuse"
14:38 2013-12-10
naive recursive algorithm
14:39 2013-12-10
you recursively call Fibonacci()
14:40 2013-12-10
very bad! exponential time
14:40 2013-12-10
recurrence relationship
14:41 2013-12-10
Fn is just the golden ratio to the nth order
14:42 2013-12-10
that general approach is called "memorization"
14:44 2013-12-10
memorized DP algorithm
14:44 2013-12-10
overlapping subproblem
14:45 2013-12-10
recursion tree
14:47 2013-12-10
fib(k) only recurses the first time it's called
* memorized calls cost O(1)
14:51 2013-12-10
number of nonmemorized calls is n!
fib(1), fib(2), ... , fib(n)
14:51 2013-12-10
DP
* memorize(remember) & reuse solutions
to subproblems that helps solve your actual problem
14:56 2013-12-10
DP == recursion + memorization
14:56 2013-12-10
running time == #subproblems * time/subproblem
14:57 2013-12-10
memorized DP algorithm
14:58 2013-12-10
bottom-up DP algorithm
14:58 2013-12-10
bottom-up DP algorithm:
* exact same computation(as the memorized DP)
* topological sort of the subproblem dependency DAG
15:07 2013-12-10
topological sort of a DAG
15:07 2013-12-10
usually it's too easy, just a for loop
15:08 2013-12-10
one thing good about this bottom up DP
is save space!
15:10 2013-12-10
we move on to shortest path
15:10 2013-12-10
Shortest Paths: δ(s, v) for any v
15:15 2013-12-10
Guessing: don't know the answer? guess
15:15 2013-12-10
...try all guesses(&take best one)
15:16 2013-12-10
DP == recursion + memorization + guessing
15:18 2013-12-10
incoming edge, outgoing edge
15:29 2013-12-10
infinite tiem on graphs with cycles
15:33 2013-12-10
subproblem dependencies should be acyclic
15:38 2013-12-10
for DP to work, for memorization to work, it's
better to be acyclic!
15:38 2013-12-10
indegree(k) // #subproblems
-----------------------------------------------
15:48 2013-12-10
start review lec 19, Shortest Path
15:48 2013-12-10
single source shortest paths: δ(s, v) for all v
15:59 2013-12-10
try all guesses // essential to DP
16:00 2013-12-10
try all guesses & take the best one!
16:01 2013-12-10
there is some hypothetical shortest path,
I don't know where goes first, so I will
guess where goes first!
16:02 2013-12-10
guess first edge approach
16:06 2013-12-10
single target shortest path
16:06 2013-12-10
guess last edge approach
16:07 2013-12-10
incoming edges to v
16:07 2013-12-10
I'm going to guess the last edge, call it u -> v
16:08 2013-12-10
I know it's one of the incoming edges to v
16:09 2013-12-10
guess all the incoming edges to v, and recursively
compute the shortest path form s -> u, then add on
the edge v
16:10 2013-12-10
δ(s, v) = min(δ(s, u) + w(u, v)), for (u, v) ∈E
δ(s, u) is a recursive call
16:13 2013-12-10
it's definitely going to be exponential
without memorization
16:15 2013-12-10
How to solve with memorization?
δ(s, v) = min(δ(s, u) + w(u, v)), for (u, v) ∈E
if δ(s, v) is in the memo table, simply return it,
otherwise, compute δ(s, v), then write into memo table
16:16 2013-12-10
I claim that memorization makes every algorithm faster!
16:17 2013-12-10
it's not that tricky, let me draw you a graph!
16:18 2013-12-10
this is an infinite algorithm
16:21 2013-12-10
infinite time on graphs with cycles
16:21 2013-12-10
DAGs: O(V + E) // V == Vertices, E == Edges
16:21 2013-12-10
running time == #subproblems * time/subproblem
16:23 2013-12-10
How many time do I spend per subproblem?
number of incoming edges to v
16:24 2013-12-10
time for subproblem δ(s, v) == indegree(v)
16:25 2013-12-10
handshaking lemma again!
16:25 2013-12-10
* subproblem dependencies should be acyclic
16:26 2013-12-10
somehow we want to take a cyclic graph &
make it acyclic!
16:27 2013-12-10
let's take a very simple cyclic graph
16:29 2013-12-10
one thing we could do is explode it into multiple layers!
16:29 2013-12-10
What is δk(s, v)?
δk(s, v) is the weight of the shortest s -> v
path that uses the <= k edges
// use at most k edges
16:32 2013-12-10
our new recurrence:
δk(s, v) = min(δk-1(s, v) + w(u, v))
for all (u, v) ∈E
16:42 2013-12-10
this is Bellman-Ford algorithm again!
--------------------------------------------
16:43 2013-12-10
start introduction to algorithm, lec 20
Dynamic Programming II: Text justification, Blackjack
16:43 2013-12-10
optimization problem
19:37 2013-12-10
DP == "careful brute force"
DP == guessing + recursion + memorization
DP == shortest paths in some DAG
19:38 2013-12-10
time == #subproblems * time/subproblem
19:38 2013-12-10
many problems really are dynamic programming in disguise
19:40 2013-12-10
dynamic programming in many case is just
computing shortest path in a DAG!
19:41 2013-12-10
amortization
19:44 2013-12-10
What are the five easy steps to Dynamic Programming?
1. define subproblems
2. guess(part of solution)
3. relate subproblem solutions // recurrence
4 recurse & memorize
or (build DP table bottom-up)
5. solve the original problem
19:48 2013-12-10
What is δk(s, v)?
it's the shortest path from s to v use at
most k edges
19:48 2013-12-10
in shortest path, let's guess what the last
edge is?
19:52 2013-12-10
it's slow, but once you have memorization,
it's fast!
19:53 2013-12-10
we want to check that the subproblem recurrence
is acyclic, in other words, it has a topological
order, so we can use topological sort
19:55 2013-12-10
Text Justification:
split text into "good" lines
20:01 2013-12-10
text == list of words
20:06 2013-12-10
dynamic programming is just a kind of
"careful brute force"
20:06 2013-12-10
what's the natrual first thing to guess?
20:08 2013-12-10
Where does the 2nd line begin? I don't know,
so guess!
20:08 2013-12-10
1. subproblems: suffixes words[i:]
2. guess: where to start the 2nd line?
3. recurrence relation
20:19 2013-12-10
total time == #subproblems * time/subproblem
20:19 2013-12-10
topological order
20:23 2013-12-10
finallly we want to check that the original
problem has been solved
20:23 2013-12-10
parent pointer:
just remember which guess was best
20:28 2013-12-10
bottom-up DP tables
20:28 2013-12-10
What is a blackjack?
20:34 2013-12-10
perfect-information blackjack
20:35 2013-12-10
win, lose, tie
---------------------------------------