Hykilpikonna
326 lines
42 KiB
HTML
326 lines
42 KiB
HTML
<!DOCTYPE html>
|
||
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||
<head>
|
||
<meta charset="utf-8" />
|
||
<meta name="generator" content="pandoc" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||
<title>8.7 Worst-Case Running Time Analysis</title>
|
||
<style>
|
||
code{white-space: pre-wrap;}
|
||
span.smallcaps{font-variant: small-caps;}
|
||
span.underline{text-decoration: underline;}
|
||
div.column{display: inline-block; vertical-align: top; width: 50%;}
|
||
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||
ul.task-list{list-style: none;}
|
||
pre > code.sourceCode { white-space: pre; position: relative; }
|
||
pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
|
||
pre > code.sourceCode > span:empty { height: 1.2em; }
|
||
code.sourceCode > span { color: inherit; text-decoration: inherit; }
|
||
div.sourceCode { margin: 1em 0; }
|
||
pre.sourceCode { margin: 0; }
|
||
@media screen {
|
||
div.sourceCode { overflow: auto; }
|
||
}
|
||
@media print {
|
||
pre > code.sourceCode { white-space: pre-wrap; }
|
||
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
|
||
}
|
||
pre.numberSource code
|
||
{ counter-reset: source-line 0; }
|
||
pre.numberSource code > span
|
||
{ position: relative; left: -4em; counter-increment: source-line; }
|
||
pre.numberSource code > span > a:first-child::before
|
||
{ content: counter(source-line);
|
||
position: relative; left: -1em; text-align: right; vertical-align: baseline;
|
||
border: none; display: inline-block;
|
||
-webkit-touch-callout: none; -webkit-user-select: none;
|
||
-khtml-user-select: none; -moz-user-select: none;
|
||
-ms-user-select: none; user-select: none;
|
||
padding: 0 4px; width: 4em;
|
||
color: #aaaaaa;
|
||
}
|
||
pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; }
|
||
div.sourceCode
|
||
{ }
|
||
@media screen {
|
||
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
|
||
}
|
||
code span.al { color: #ff0000; font-weight: bold; } /* Alert */
|
||
code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
|
||
code span.at { color: #7d9029; } /* Attribute */
|
||
code span.bn { color: #40a070; } /* BaseN */
|
||
code span.bu { } /* BuiltIn */
|
||
code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
|
||
code span.ch { color: #4070a0; } /* Char */
|
||
code span.cn { color: #880000; } /* Constant */
|
||
code span.co { color: #60a0b0; font-style: italic; } /* Comment */
|
||
code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
|
||
code span.do { color: #ba2121; font-style: italic; } /* Documentation */
|
||
code span.dt { color: #902000; } /* DataType */
|
||
code span.dv { color: #40a070; } /* DecVal */
|
||
code span.er { color: #ff0000; font-weight: bold; } /* Error */
|
||
code span.ex { } /* Extension */
|
||
code span.fl { color: #40a070; } /* Float */
|
||
code span.fu { color: #06287e; } /* Function */
|
||
code span.im { } /* Import */
|
||
code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
|
||
code span.kw { color: #007020; font-weight: bold; } /* Keyword */
|
||
code span.op { color: #666666; } /* Operator */
|
||
code span.ot { color: #007020; } /* Other */
|
||
code span.pp { color: #bc7a00; } /* Preprocessor */
|
||
code span.sc { color: #4070a0; } /* SpecialChar */
|
||
code span.ss { color: #bb6688; } /* SpecialString */
|
||
code span.st { color: #4070a0; } /* String */
|
||
code span.va { color: #19177c; } /* Variable */
|
||
code span.vs { color: #4070a0; } /* VerbatimString */
|
||
code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
|
||
</style>
|
||
<link rel="stylesheet" href="../tufte.css" />
|
||
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js" type="text/javascript"></script>
|
||
<!--[if lt IE 9]>
|
||
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
|
||
<![endif]-->
|
||
</head>
|
||
<body>
|
||
<div style="display:none">
|
||
\(
|
||
\newcommand{\NOT}{\neg}
|
||
\newcommand{\AND}{\wedge}
|
||
\newcommand{\OR}{\vee}
|
||
\newcommand{\XOR}{\oplus}
|
||
\newcommand{\IMP}{\Rightarrow}
|
||
\newcommand{\IFF}{\Leftrightarrow}
|
||
\newcommand{\TRUE}{\text{True}\xspace}
|
||
\newcommand{\FALSE}{\text{False}\xspace}
|
||
\newcommand{\IN}{\,{\in}\,}
|
||
\newcommand{\NOTIN}{\,{\notin}\,}
|
||
\newcommand{\TO}{\rightarrow}
|
||
\newcommand{\DIV}{\mid}
|
||
\newcommand{\NDIV}{\nmid}
|
||
\newcommand{\MOD}[1]{\pmod{#1}}
|
||
\newcommand{\MODS}[1]{\ (\text{mod}\ #1)}
|
||
\newcommand{\N}{\mathbb N}
|
||
\newcommand{\Z}{\mathbb Z}
|
||
\newcommand{\Q}{\mathbb Q}
|
||
\newcommand{\R}{\mathbb R}
|
||
\newcommand{\C}{\mathbb C}
|
||
\newcommand{\cA}{\mathcal A}
|
||
\newcommand{\cB}{\mathcal B}
|
||
\newcommand{\cC}{\mathcal C}
|
||
\newcommand{\cD}{\mathcal D}
|
||
\newcommand{\cE}{\mathcal E}
|
||
\newcommand{\cF}{\mathcal F}
|
||
\newcommand{\cG}{\mathcal G}
|
||
\newcommand{\cH}{\mathcal H}
|
||
\newcommand{\cI}{\mathcal I}
|
||
\newcommand{\cJ}{\mathcal J}
|
||
\newcommand{\cL}{\mathcal L}
|
||
\newcommand{\cK}{\mathcal K}
|
||
\newcommand{\cN}{\mathcal N}
|
||
\newcommand{\cO}{\mathcal O}
|
||
\newcommand{\cP}{\mathcal P}
|
||
\newcommand{\cQ}{\mathcal Q}
|
||
\newcommand{\cS}{\mathcal S}
|
||
\newcommand{\cT}{\mathcal T}
|
||
\newcommand{\cV}{\mathcal V}
|
||
\newcommand{\cW}{\mathcal W}
|
||
\newcommand{\cZ}{\mathcal Z}
|
||
\newcommand{\emp}{\emptyset}
|
||
\newcommand{\bs}{\backslash}
|
||
\newcommand{\floor}[1]{\left \lfloor #1 \right \rfloor}
|
||
\newcommand{\ceil}[1]{\left \lceil #1 \right \rceil}
|
||
\newcommand{\abs}[1]{\left | #1 \right |}
|
||
\newcommand{\xspace}{}
|
||
\newcommand{\proofheader}[1]{\underline{\textbf{#1}}}
|
||
\)
|
||
</div>
|
||
<header id="title-block-header">
|
||
<h1 class="title">8.7 Worst-Case Running Time Analysis</h1>
|
||
</header>
|
||
<section>
|
||
<p>In Section 8.3, we saw how to use asymptotic notation to characterize the <em>rate of growth</em> of the number of “basic operations” as a way of analyzing the running time of an algorithm. This approach allows us to ignore details of the computing environment in which the algorithm is run, and machine- and language-dependent implementations of primitive operations, and instead characterize the relationship between the input size and number of basic operations performed.</p>
|
||
<p>However, this focus on just the input size is a little too restrictive. Even though we can define input size differently for each algorithm we analyze, we tend not to stray too far from the “natural” definitions (e.g., length of list). In practice, though, algorithms often depend on the actual value of the input, not just its size. For example, consider the following function, which searches for an even number in a list of integers.<label for="sn-0" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-0" class="margin-toggle"/><span class="sidenote"> This is very similar to how the <code>in</code> operator is implemented for Python lists.</span></p>
|
||
<div class="sourceCode" id="cb1"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1"></a><span class="kw">def</span> has_even(numbers: <span class="bu">list</span>[<span class="bu">int</span>]) <span class="op">-></span> <span class="bu">bool</span>:</span>
|
||
<span id="cb1-2"><a href="#cb1-2"></a> <span class="co">"""Return whether numbers contains an even element."""</span></span>
|
||
<span id="cb1-3"><a href="#cb1-3"></a> <span class="cf">for</span> number <span class="kw">in</span> numbers:</span>
|
||
<span id="cb1-4"><a href="#cb1-4"></a> <span class="cf">if</span> number <span class="op">%</span> <span class="dv">2</span> <span class="op">==</span> <span class="dv">0</span>:</span>
|
||
<span id="cb1-5"><a href="#cb1-5"></a> <span class="cf">return</span> <span class="va">True</span></span>
|
||
<span id="cb1-6"><a href="#cb1-6"></a> <span class="cf">return</span> <span class="va">False</span></span></code></pre></div>
|
||
<p>Because this function returns as soon as it finds an even number in the list, its running time is not necessarily proportional to the length of the input list.</p>
|
||
<p><strong>The running time of a function can vary even when the input size is fixed.</strong> Or using the notation we learned earlier this chapter, the inputs in <span class="math inline">\(\cI_{has\_even, 10}\)</span> do <em>not</em> all have the same runtime. The question “what is <em>the</em> running time of <code>has_even</code> on an input of length <span class="math inline">\(n\)</span>?” does not make sense, as for a given input the runtime depends not just on its length but on which of its elements are even. We illustrate in the following plot, which shows the results of using <code>timeit</code> to measure the running time of <code>has_evens</code> on randomly-chosen lists. While every timing experiment has some inherent uncertainty in the results, the spread of running times cannot be explained by that alone!</p>
|
||
<p><img src="images/has_evens_plotly.png" style="width:100.0%" alt="Running time plot for has_evens." /><br />
|
||
</p>
|
||
<p>Because our asymptotic notation is used to describe the growth rate of <em>functions</em>, we cannot use it to describe the growth of a whole range of values with respect to increasing input sizes. A natural approach to fix this problem is to focus on the <em>maximum</em> of this range, which corresponds to the <em>slowest</em> the algorithm could run for a given input size.</p>
|
||
<div class="definition" data-terms="worst-case runtime">
|
||
<p>Let <code>func</code> be a program. We define the function <span class="math inline">\(WC_{func}: \N \to \N\)</span>, called the <strong>worst-case running time function of <code>func</code></strong>, as follows:<label for="sn-1" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-1" class="margin-toggle"/><span class="sidenote">Here, “running time” is measured in exact number of basic operations. We are taking the maximum of a set of numbers, <em>not</em> a set of asymptotic expressions.</span> <span class="math display">\[
|
||
WC_{func}(n) = \max \big\{ \text{running time of executing $func(x)$} \mid x \in \cI_{func, n} \big\}
|
||
\]</span></p>
|
||
</div>
|
||
<p>Note that <span class="math inline">\(WC_{func}\)</span> is a function, not a (constant) number: it returns the maximum possible running time for an input of size <span class="math inline">\(n\)</span>, for every natural number <span class="math inline">\(n\)</span>. And because it is a function, we can use asymptotic notation to describe it, saying things like “the worst-case running time of this function is <span class="math inline">\(\Theta(n^2)\)</span>.”</p>
|
||
<p>The goal of a <em>worst-case runtime analysis</em> for <code>func</code> is to find an elementary function <span class="math inline">\(f\)</span> such that <span class="math inline">\(WC_{func} \in \Theta(f)\)</span>.</p>
|
||
<p>However, it takes a bit more work to obtain tight bounds on a worst-case running time than on the runtime functions of the previous section. It is difficult to compute the <em>exact maximum</em> number of basic operations performed by this algorithm for every input size, which requires that we identify an input for each input size, count its maximum number of basic operations, and then prove that every input of this size takes at most this number of operations. Instead, we will generally take a two-pronged approach: proving matching <em>upper</em> and <em>lower bounds</em> on the worst-case running time of our algorithm.</p>
|
||
<h3 id="upper-bounds-on-the-worst-case-runtime">Upper bounds on the worst-case runtime</h3>
|
||
<div class="definition" data-terms="upper bound on worst-case runtime">
|
||
<p>Let <code>func</code> be a program, and <span class="math inline">\(WC_{func}\)</span> its worst-case runtime function. We say that a function <span class="math inline">\(f: \N \to \R^{\geq 0}\)</span> is an <strong>upper bound on the worst-case runtime</strong> when <span class="math inline">\(WC_{func} \in \cO(f)\)</span>.</p>
|
||
</div>
|
||
<p>To get some intuition about what an upper bound on the worst-case running means, suppose we use absolute dominance rather than Big-O. In this case, there’s a very intuitive way to expand the phrase “<span class="math inline">\(WC_{func}\)</span> is absolutely dominated by <span class="math inline">\(f\)</span>”:</p>
|
||
<p><span class="math display">\[\begin{align*}
|
||
&\forall n \in \N,~ WC_{func}(n) \leq f(n) \\
|
||
\Longleftrightarrow \, &\forall n \in \N,~ \max \big\{ \text{running time of executing $func(x)$} \mid x \in \cI_{func, n} \big\} \leq f(n) \\
|
||
\Longleftrightarrow \, &\forall n \in \N,~ \forall x \in \cI_{func, n},~ \text{running time of executing $func(x)$} \leq f(n)
|
||
\end{align*}\]</span></p>
|
||
<p>The last line comes from the fact that if we know the maximum of a set of numbers is less than some value <span class="math inline">\(K\)</span>, then <em>all</em> numbers in that set must be less than <span class="math inline">\(K\)</span>. Thus an upper bound on the worst-case runtime is equivalent to an upper bound on the runtimes of <em>all</em> inputs.</p>
|
||
<p>Now when we apply the definition of Big-O instead of absolute dominance, we get the following translation of <span class="math inline">\(WC_{func} \in \cO(f)\)</span>:</p>
|
||
<p><span class="math display">\[
|
||
\exists c, n_0 \in \R^+,~ \forall n \in \N,~ n \geq n_0 \Rightarrow \big(\forall x \in \cI_{func, n},~ \text{running time of executing $func(x)$} \leq c \cdot f(n) \big)
|
||
\]</span></p>
|
||
</div>
|
||
<p>To approach an analysis of an upper bound on the worst-case, we typically find a function <span class="math inline">\(g\)</span> such that <span class="math inline">\(WC_{func}\)</span> is absolutely dominated by <span class="math inline">\(g\)</span>, and then find a simple function <span class="math inline">\(f\)</span> such that <span class="math inline">\(g \in \cO(f)\)</span>. But how do we find such a <span class="math inline">\(g\)</span>? And what does it mean to upper bound <em>all</em> runtimes of a given input size? We’ll illustrate the technique in our next example.</p>
|
||
<div id="example:worst_case_upper_bound" class="example">
|
||
<p>Find an asymptotic upper bound on the worst-case running time of <code>has_even</code>.</p>
|
||
<div class="discussion">
|
||
<p>The intuitive translation using absolute dominance is usually enough for an upper bound analysis. In particular, the <span class="math inline">\(\forall n \in \N,~ \forall x \in \cI_{func, n}\)</span> begins with two universal quantifiers, and just knowing this alone should anticipate how we’ll start our proof, using the same techniques of proof we learned earlier!</p>
|
||
</div>
|
||
<div class="analysis">
|
||
<p><em>(Upper bound on worst-case)</em></p>
|
||
<p>First, let <span class="math inline">\(n \in \N\)</span> and let <code>numbers</code> be an <em>arbitrary</em> list of length <span class="math inline">\(n\)</span>.</p>
|
||
<p>Now we’ll analyse the running time of <code>has_even</code>, except we <em>can’t</em> assume anything about the values inside <code>numbers</code>, because it’s an aribtrary list. But we can still find an <em>upper bound</em> on the running time:</p>
|
||
<ul>
|
||
<li><p>The loop (<code>for number in numbers</code>) iterates <em>at most</em> <span class="math inline">\(n\)</span> times. Each loop iteration counts as a single step (because it is constant time), so the loop takes <em>at most</em> <span class="math inline">\(n \cdot 1 = n\)</span> steps in total.</p></li>
|
||
<li><p>The <code>return False</code> statement (if it is executed) counts as <span class="math inline">\(1\)</span> basic operation.</p></li>
|
||
</ul>
|
||
<p>Therefore the running time is <em>at most <span class="math inline">\(n + 1\)</span></em>, and <span class="math inline">\(n + 1 \in \cO(n)\)</span>. So we can conclude that the worst-case running time of <code>has_even</code> is <span class="math inline">\(\cO(n)\)</span>.</p>
|
||
</div>
|
||
</div>
|
||
<p>Note that we did <em>not</em> prove that <code>has_even(numbers)</code> takes exactly <span class="math inline">\(n + 1\)</span> basic operations for an arbitrary input <code>numbers</code> (this is false); we only proved an <em>upper bound</em> on the number of operations. And in fact, we don’t even care that much about the exact number: what we ultimately care about is the asymptotic growth rate, which is linear for <span class="math inline">\(n + 1\)</span>. This allowed us to conclude that the worst-case running time of <code>has_even</code> is <span class="math inline">\(\cO(n)\)</span>.</p>
|
||
<p>But because we calculated an upper bound rather than an exact number of steps, we can only conclude a Big-O, not Theta bound: we don’t yet know that this upper bound is tight.<label for="sn-2" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-2" class="margin-toggle"/><span class="sidenote">If this is surprising, note that we could have done the above proof but replaced <span class="math inline">\(n+1\)</span> by <span class="math inline">\(5000n + 110\)</span> and it would still have been mathematically valid.</span></p>
|
||
<h3 id="lower-bounds-on-the-worst-case-runtime">Lower bounds on the worst-case runtime</h3>
|
||
<p>So how do we prove our upper bound is tight? Since we’ve just shown that <span class="math inline">\(WC_{has\_even}(n) \in \cO(n)\)</span>, we need to prove the corresponding lower bound <span class="math inline">\(WC_{has\_even}(n) \in \Omega(n)\)</span>. But what does it mean to prove a lower bound on the maximum of a set of numbers? Suppose we have a set of numbers <span class="math inline">\(S\)</span>, and say that “the maximum of <span class="math inline">\(S\)</span> is at least <span class="math inline">\(50\)</span>.” This doesn’t tell us what the maximum of <span class="math inline">\(S\)</span> actually is, but it does give us one piece of information: there has to be a number in <span class="math inline">\(S\)</span> which is at least <span class="math inline">\(50\)</span>.</p>
|
||
<p>The key insight is that the converse is also true—if I tell you that <span class="math inline">\(S\)</span> contains the number <span class="math inline">\(50\)</span>, then you can conclude that the maximum of <span class="math inline">\(S\)</span> is at least <span class="math inline">\(50\)</span>. <span class="math display">\[\max(S) \geq 50 \IFF (\exists x \in S,~ x \geq 50)\]</span> Using this idea, we’ll give a formal definition for a lower bound on the worst-case runtime of an algorithm.</p>
|
||
<div class="definition" data-terms="lower bound on worst-case runtime">
|
||
<p>Let <code>func</code> be a program, and <span class="math inline">\(WC_{func}\)</span> is worst-case runtime function. We say that a function <span class="math inline">\(f: \N \to \R^{\geq 0}\)</span> is a <strong>lower bound on the worst-case runtime</strong> when <span class="math inline">\(WC_{func} \in \Omega(f)\)</span>.</p>
|
||
<p>In an analogous fashion to the upper bound, we unpack this definition first by using absolute dominance: <span class="math display">\[\begin{align*}
|
||
& \forall n \in \N,~ WC_{func}(n) \geq f(n) \\
|
||
\Longleftrightarrow \, &\forall n \in \N,~ \max \big\{ \text{running time of executing $func(x)$} \mid x \in \cI_{func, n} \big\} \geq f(n) \\
|
||
\Longleftrightarrow \, &\forall n \in \N,~ \exists x \in \cI_{func, n},~ \text{running time of executing $func(x)$} \geq f(n)
|
||
\end{align*}\]</span></p>
|
||
<p>And then using Omega:</p>
|
||
<p><span class="math display">\[
|
||
\exists c, n_0 \in \R^+,~ \forall n \in \N,~ n \geq n_0 \Rightarrow
|
||
\big(\exists x \in \cI_{func, n},~ \text{running time of executing $func(x)$} \geq c \cdot f(n) \big)
|
||
\]</span></p>
|
||
</div>
|
||
<p>Remarkably, the crucial difference between this definition and the one for upper bounds is a change of quantifier: now the input <span class="math inline">\(x\)</span> is existentially quantified, meaning we get to pick it. Or really, our goal is to find an <strong>input family</strong>—a <em>set</em> of inputs, one per input size <span class="math inline">\(n\)</span>—whose runtime is asymptotically larger than our target lower bound.</p>
|
||
<p>For example, in <code>has_even</code> we want to prove that the worst-case running time is <span class="math inline">\(\Omega(n)\)</span> to match the <span class="math inline">\(\cO(n)\)</span> upper bound, and so we want to find and input family where the number of steps taken is <span class="math inline">\(\Omega(n)\)</span>. Let’s do that now.</p>
|
||
<div id="example:worst_case_lower_bound" class="example">
|
||
<p>Find an asymptotic lower bound on the worst-case running time of <code>has_even</code>.</p>
|
||
<div class="discussion">
|
||
<p>Again, we’ll just remind you of the quantifiers from the intuitive “absolute dominance” version of the lower bound definition: <span class="math inline">\(\forall n \in \N,~ \exists x \in \cI_{n}\)</span>. This will inform how we start our proof.</p>
|
||
</div>
|
||
<div class="analysis">
|
||
<p><em>(Lower bound on worst-case)</em></p>
|
||
<p>Let <span class="math inline">\(n \in \N\)</span>. Let <code>numbers</code> be the list of length <span class="math inline">\(n\)</span> consisting of all <span class="math inline">\(1\)</span>’s. Now we’ll analyse the (exact) running time of <code>has_even</code> on this input.</p>
|
||
<p>In this case, the <code>if</code> condition in the loop is always false, so the loop never stops early. Therefore it iterates exactly <span class="math inline">\(n\)</span> times (once per item in the list), with each iteration taking one step.</p>
|
||
<p>Finally, the <code>return False</code> statement executes, which is one step. So the total number of steps for this input is <span class="math inline">\(n + 1\)</span>, which is <span class="math inline">\(\Omega(n)\)</span>.</p>
|
||
</div>
|
||
</div>
|
||
<h3 id="putting-it-all-together">Putting it all together</h3>
|
||
<p>Finally, we can combine our upper and lower bounds on <span class="math inline">\(WC_{has\_even}\)</span> to obtain a tight asymptotic bound.</p>
|
||
<div class="example">
|
||
<p>Find a <em>tight</em> bound on the worst-case running time of <code>has_even</code>.</p>
|
||
<div class="analysis">
|
||
<p>Since we’ve proved that <span class="math inline">\(WC_{has\_even}\)</span> is <span class="math inline">\(\cO(n)\)</span> and <span class="math inline">\(\Omega(n)\)</span>, it is <span class="math inline">\(\Theta(n)\)</span>.</p>
|
||
</div>
|
||
</div>
|
||
<p>To summarize, to obtain a tight bound on the worst-case running time of a function, we need to do two things:</p>
|
||
<ul>
|
||
<li><p>Use the properties of the code to obtain an <em>asymptotic upper bound</em> on the worst-case running time. We would say something like <span class="math inline">\(WC_{func} \in \cO(f)\)</span>.</p></li>
|
||
<li><p>Find a family of inputs whose running time is <span class="math inline">\(\Omega(f)\)</span>.<label for="sn-3" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-3" class="margin-toggle"/><span class="sidenote"> Almost always we find an input family whose running time is <span class="math inline">\(\Theta(f)\)</span>, but strictly speaking only <span class="math inline">\(\Omega(f)\)</span> is required.</span> This will prove that <span class="math inline">\(WC_{func} \in \Omega(f)\)</span>.</p></li>
|
||
<li><p>After showing that <span class="math inline">\(WC_{func} \in \cO(f)\)</span> and <span class="math inline">\(WC_{func} \in \Omega(f)\)</span>, we can conclude that <span class="math inline">\(WC_f \in \Theta(f)\)</span>.</p></li>
|
||
</ul>
|
||
<h3 id="a-note-about-best-case-runtime">A note about best-case runtime</h3>
|
||
<p>In this section, we focused on worst-case runtime, the result of taking the <em>maximum</em> runtime for every input size. It is also possible to define a best-case runtime function by taking the minimum possible runtimes, and obtain tight bounds on the best case through an analysis that is completely analogous to the one we just performed. In practice, however, the best-case runtime of an algorithm is usually not as useful to know—we care far more about knowing just how <em>slow</em> an algorithm is than how fast it can be.</p>
|
||
<h2 id="early-returning-in-python-built-ins">Early returning in Python built-ins</h2>
|
||
<p>We’ve encountered a few different Python functions and methods whose running time depends on more than just the size of their inputs. We alluded to one at the start of this chapter: the list search operation using the keyword <code>in</code>:</p>
|
||
<div class="sourceCode" id="cb2"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1"></a><span class="op">>>></span> lst <span class="op">=</span> <span class="bu">list</span>(<span class="bu">range</span>(<span class="dv">0</span>, <span class="dv">1000000</span>))</span>
|
||
<span id="cb2-2"><a href="#cb2-2"></a><span class="op">>>></span> timeit.timeit(<span class="st">'0 in lst'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||
<span id="cb2-3"><a href="#cb2-3"></a><span class="fl">8.299997716676444e-06</span></span>
|
||
<span id="cb2-4"><a href="#cb2-4"></a><span class="op">>>></span> timeit.timeit(<span class="st">'-1 in lst'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||
<span id="cb2-5"><a href="#cb2-5"></a><span class="fl">0.17990550000104122</span></span></code></pre></div>
|
||
<p>In the first <code>timeit</code> expression, <code>0</code> appears as the first element of <code>lst</code>, and so is found immediately when the search occurs. In the second, <code>-1</code> does not appear in <code>lst</code> at all, and so all one-million elements of <code>lst</code> must be checked, resulting in a running-time that is proportional to the length of the list. <em>The worst-case running time of the <code>in</code> operation for lists is <span class="math inline">\(\Theta(n)\)</span>, where <span class="math inline">\(n\)</span> is the length of the list.</em></p>
|
||
<p>We have also seen two more functions that are implemented using an early return: <code>any</code> and <code>all</code>. Because <code>any</code> searches for a single <code>True</code> in a collection, it stops the first time it finds one. Similarly, because <code>all</code> requires that all elements of a collection be <code>True</code>, it stops the first time it finds a <code>False</code> value.</p>
|
||
<div class="sourceCode" id="cb3"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1"></a><span class="op">>>></span> all_trues <span class="op">=</span> [<span class="va">True</span>] <span class="op">*</span> <span class="dv">1000000</span></span>
|
||
<span id="cb3-2"><a href="#cb3-2"></a><span class="op">>>></span> all_falses <span class="op">=</span> [<span class="va">False</span>] <span class="op">*</span> <span class="dv">1000000</span></span>
|
||
<span id="cb3-3"><a href="#cb3-3"></a><span class="op">>>></span> timeit.timeit(<span class="st">'any(all_trues)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||
<span id="cb3-4"><a href="#cb3-4"></a><span class="fl">8.600000001024455e-06</span></span>
|
||
<span id="cb3-5"><a href="#cb3-5"></a><span class="op">>>></span> timeit.timeit(<span class="st">'any(all_falses)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||
<span id="cb3-6"><a href="#cb3-6"></a><span class="fl">0.10643419999905745</span></span>
|
||
<span id="cb3-7"><a href="#cb3-7"></a><span class="op">>>></span> timeit.timeit(<span class="st">'all(all_trues)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||
<span id="cb3-8"><a href="#cb3-8"></a><span class="fl">0.10217570000168053</span></span>
|
||
<span id="cb3-9"><a href="#cb3-9"></a><span class="op">>>></span> timeit.timeit(<span class="st">'all(all_falses)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||
<span id="cb3-10"><a href="#cb3-10"></a><span class="fl">6.300000677583739e-06</span></span></code></pre></div>
|
||
<p>So in the above example:</p>
|
||
<ul>
|
||
<li><code>any(all_trues)</code> returns <code>True</code> immediately after checking the first list element.</li>
|
||
<li><code>any(all_falses)</code> returns <code>False</code> only after checking all one-million list elements.</li>
|
||
<li><code>all(all_trues)</code> returns <code>True</code> only after checking all one-million list elements.</li>
|
||
<li><code>all(all_falses)</code> returns <code>False</code> immediately after checking the first list element.</li>
|
||
</ul>
|
||
<p>So <code>any</code> and <code>all</code> have a worst-case running time of <span class="math inline">\(\Theta(n)\)</span>, where <span class="math inline">\(n\)</span> is the size of the input collection. But in practice they can be much faster if they encounter the “right” boolean value early on!</p>
|
||
<h3 id="any-all-and-comprehensions"><code>any</code>, <code>all</code>, and comprehensions</h3>
|
||
<p>There is one subtlety that often catches students by surprise when they attempt to call <code>any</code>/<code>all</code> on a comprehension and expect a quick result. Let’s see a simple example:</p>
|
||
<div class="sourceCode" id="cb4"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb4-1"><a href="#cb4-1"></a><span class="op">>>></span> timeit.timeit(<span class="st">'any([x == 0 for x in range(0, 1000000)])'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||
<span id="cb4-2"><a href="#cb4-2"></a><span class="fl">0.7032962000012049</span></span></code></pre></div>
|
||
<p>That’s a lot slower than we would expect, given that the first element checked is <code>x = 0</code>! The result is similar if we try to use a set comprehension instead of a list comprehension:</p>
|
||
<div class="sourceCode" id="cb5"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb5-1"><a href="#cb5-1"></a><span class="op">>>></span> timeit.timeit(<span class="st">'any({x == 0 for x in range(0, 1000000)})'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||
<span id="cb5-2"><a href="#cb5-2"></a><span class="fl">0.6538308000017423</span></span></code></pre></div>
|
||
<p>The subtlety here is that in both cases, <em>the full comprehension is evaluated before <code>any</code> is called</em>. As we discussed in <a href="05-more-runtime.html">8.5 Analyzing Comprehensions and While Loops</a>, the running time of evaluating a comprehension is proportional to the size of the source collection of the comprehension—in our example, that’s <code>range(0, 1000000)</code>, which contains one-million numbers.</p>
|
||
<p>But all is not lost! In practice, Python programmers <em>do</em> use <code>any</code>/<code>all</code> with comprehensions, but they do so by writing the comprehension expression in the function call without any surrounding square brackets or curly braces:</p>
|
||
<div class="sourceCode" id="cb6"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb6-1"><a href="#cb6-1"></a><span class="op">>>></span> <span class="bu">any</span>(x <span class="op">==</span> <span class="dv">0</span> <span class="cf">for</span> x <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, <span class="dv">1000000</span>))</span>
|
||
<span id="cb6-2"><a href="#cb6-2"></a><span class="va">True</span></span></code></pre></div>
|
||
<p>This is called a <strong>generator comprehension</strong>, and is used to produce a special Python collection data type called a <strong>generator</strong>. We won’t use generators or generator comprehensions very much at all in this course, but what we want you to know about them here is that unlike set/list comprehensions, generator comprehensions do not evaluate their elements all at once, but instead only when they are needed by the function being called. This means that our above <code>any</code> call achieves the fast running time we initially expected:</p>
|
||
<div class="sourceCode" id="cb7"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb7-1"><a href="#cb7-1"></a><span class="op">>>></span> timeit.timeit(<span class="st">'any(x == 0 for x in range(0, 1000000))'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||
<span id="cb7-2"><a href="#cb7-2"></a><span class="fl">4.050000279676169e-05</span></span></code></pre></div>
|
||
<p>Now, only the <code>x = 0</code> value from the generator comprehension gets evaluated; none of the other possible values (<code>x = 1, 2, ..., 999999</code>) are ever checked by the <code>any</code> call!</p>
|
||
<h2 id="dont-assume-bounds-are-tight">Don’t assume bounds are tight!</h2>
|
||
<p>It is likely unsatisfying to hear that upper and lower bounds really are distinct things that must be computed separately. Our intuition here pulls us towards the bounds being “obviously” the same, but this is really a side effect of the examples we have studied so far in this course being rather straightforward. But this won’t always be the case: the study of more complex algorithms and data structures exhibits quite a few situations where obtaining an upper bound on the running time involves a completely different analysis than the lower bound.</p>
|
||
<p>Let’s look at one such example that deals with manipulating strings.</p>
|
||
<div class="example">
|
||
<p>We say that a string is a <strong>palindrome</strong> when it can be read the same forwards and backwards; example of palindromes are “abba”, “racecar”, and “z”.<label for="sn-4" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-4" class="margin-toggle"/><span class="sidenote">Every string of length 1 is a palindrome.</span> We say that a string <span class="math inline">\(s_1\)</span> is a <strong>prefix</strong> of another string <span class="math inline">\(s_2\)</span> when <span class="math inline">\(s_1\)</span> is a substring of <span class="math inline">\(s_2\)</span> that starts at index 0 of <span class="math inline">\(s_2\)</span>. For example, the string “abc” is a prefix of “abcdef”.</p>
|
||
<p>The algorithm below takes a non-empty string as input, and returns the length of the longest prefix of that string that is a palindrome. For example, the string “attack” has two non-empty prefixes that are palindromes, “a” and “atta”, and so our algorithm will return 4.</p>
|
||
<div class="sourceCode" id="cb8"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb8-1"><a href="#cb8-1"></a><span class="kw">def</span> palindrome_prefix(s: <span class="bu">str</span>) <span class="op">-></span> <span class="bu">int</span>:</span>
|
||
<span id="cb8-2"><a href="#cb8-2"></a> n <span class="op">=</span> <span class="bu">len</span>(s)</span>
|
||
<span id="cb8-3"><a href="#cb8-3"></a> <span class="cf">for</span> prefix_length <span class="kw">in</span> <span class="bu">range</span>(n, <span class="dv">0</span>, <span class="op">-</span><span class="dv">1</span>): <span class="co"># goes from n down to 1</span></span>
|
||
<span id="cb8-4"><a href="#cb8-4"></a> <span class="co"># Check whether s[0:prefix_length] is a palindrome</span></span>
|
||
<span id="cb8-5"><a href="#cb8-5"></a> is_palindrome <span class="op">=</span> <span class="bu">all</span>(s[i] <span class="op">==</span> s[prefix_length <span class="op">-</span> <span class="dv">1</span> <span class="op">-</span> i]</span>
|
||
<span id="cb8-6"><a href="#cb8-6"></a> <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, prefix_length))</span>
|
||
<span id="cb8-7"><a href="#cb8-7"></a></span>
|
||
<span id="cb8-8"><a href="#cb8-8"></a> <span class="co"># If a palindrome prefix is found, return the current length.</span></span>
|
||
<span id="cb8-9"><a href="#cb8-9"></a> <span class="cf">if</span> is_palindrome:</span>
|
||
<span id="cb8-10"><a href="#cb8-10"></a> <span class="cf">return</span> prefix_length</span></code></pre></div>
|
||
<p>There are a few interesting details to note about this algorithm:</p>
|
||
<ul>
|
||
<li>The for loop iterable is <code>range(n, 0, -1)</code>—the third argument <code>-1</code> causes the loop variable to <em>start</em> at <code>n</code> and decrease by 1 at each iteration. In other words, this loop is checking the possible prefixes starting with the longest prefix (length <code>n</code>) and working its way to the shortest prefix (length 1).</li>
|
||
<li>The call to <code>all</code> checks pairs of characters starting at either end of the current prefix. It uses a <em>generator comprehension</em> (like we discussed above) so that it can stop early as soon as it encounters a mismatch (i.e., when <code>s[i] != s[prefix_length - 1 - i]</code>).</li>
|
||
<li>Even though the only return statement is inside the <code>for</code> loop, this algorithm is guaranteed to find a palindrome prefix, since the first letter of <code>s</code> by itself is a palindrome.</li>
|
||
</ul>
|
||
<p>The code presented here is structurally simple. Indeed, it is not too hard to show that the worst-case runtime of this function is <span class="math inline">\(\cO(n^2)\)</span>, where <span class="math inline">\(n\)</span> is the length of the input string. What is harder, however, is showing that the worst-case runtime is <span class="math inline">\(\Omega(n^2)\)</span>. To do so, we must find an input family whose runtime is <span class="math inline">\(\Omega(n^2)\)</span>. There are two points in the code that can lead to fewer than the maximum loop iterations occurring, and we want to find an input family that avoids both of these.</p>
|
||
<p>The difficulty is that these two points are caused by different types of inputs! The call to <code>all</code> can stop as soon as the algorithm detects that a prefix is <em>not</em> a palindrome, while the <code>return</code> statement occurs when the algorithm has determined that a prefix <em>is</em> a palindrome! To make this tension more explicit, let’s consider two extreme input families that seem plausible at first glance, but which do not have a runtime that is <span class="math inline">\(\Omega(n^2)\)</span>.</p>
|
||
<ul>
|
||
<li>The entire string <span class="math inline">\(s\)</span> is a palindrome of length <span class="math inline">\(n\)</span>. In this case, in the first iteration of the loop, the entire string is checked. The <code>all</code> call checks all pairs of characters, but unfortunately this means that <code>is_palindrome = True</code>, and the loop returns during its very first iteration. Since the <code>all</code> call takes <span class="math inline">\(n\)</span> steps, this input family takes <span class="math inline">\(\Theta(n)\)</span> time to run.</li>
|
||
<li>The entire string <span class="math inline">\(s\)</span> consists of <span class="math inline">\(n\)</span> different letters. In this case, the only palindrome prefix is just the first letter of <span class="math inline">\(s\)</span> itself. This means that the loop will run for all <span class="math inline">\(n\)</span> iterations, only returning in its last iteration (when <code>prefix_length == 1</code>). However, the <code>all</code> call will always stop after just one step, since it starts by comparing the first letter of <span class="math inline">\(s\)</span> with another letter, which is guaranteed to be different by our choice of input family. This again leads to a <span class="math inline">\(\Theta(n)\)</span> running time.</li>
|
||
</ul>
|
||
<p>The key idea is that we want to choose an input family that <em>doesn’t</em> contain a long palindrome (so the loop runs for many iterations), but whose prefixes are close to being palindromes like palindromes (so the <code>all</code> call checks many pairs of letters). Let <span class="math inline">\(n \in \Z^+\)</span>. We define the input <span class="math inline">\(s_n\)</span> as follows:</p>
|
||
<ul>
|
||
<li><span class="math inline">\(s_n[\ceil{n/2}] = b\)</span></li>
|
||
<li>Every other character in <span class="math inline">\(s_n\)</span> is equal to <span class="math inline">\(a\)</span>.</li>
|
||
</ul>
|
||
<p>For example, <span class="math inline">\(s_4 = aaba\)</span> and <span class="math inline">\(s_{11} = aaaaaabaaa\)</span>. Note that <span class="math inline">\(s_n\)</span> is very close to being a palindrome: if that single character <span class="math inline">\(b\)</span> were changed to an <span class="math inline">\(a\)</span>, then <span class="math inline">\(s_n\)</span> would be the all-<span class="math inline">\(a\)</span>’s string, which is certainly a palindrome. But by making the centre character a <span class="math inline">\(b\)</span>, we not only ensure that the longest palindrome of <span class="math inline">\(s_n\)</span> has length roughly <span class="math inline">\(n/2\)</span> (so the loop iterates roughly <span class="math inline">\(n/2\)</span> times), but also that the “outer” characters of each prefix of <span class="math inline">\(s_n\)</span> containing more than <span class="math inline">\(n/2\)</span> characters are all the same (so the <code>all</code> call checks many pairs to find the mismatch between <span class="math inline">\(a\)</span> and <span class="math inline">\(b\)</span>). It turns out that this input family does indeed have an <span class="math inline">\(\Omega(n^2)\)</span> runtime! We’ll leave the details as an exercise.</p>
|
||
</div>
|
||
</section>
|
||
<footer>
|
||
<a href="https://www.teach.cs.toronto.edu/~csc110y/fall/notes/">CSC110 Course Notes Home</a>
|
||
</footer>
|
||
</body>
|
||
</html>
|