deploy
This commit is contained in:
Hykilpikonna
@@ -0,0 +1,282 @@
|
||||
<!DOCTYPE html>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="generator" content="pandoc" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||||
<title>8.1 An Introduction to Running Time</title>
|
||||
<style>
|
||||
code{white-space: pre-wrap;}
|
||||
span.smallcaps{font-variant: small-caps;}
|
||||
span.underline{text-decoration: underline;}
|
||||
div.column{display: inline-block; vertical-align: top; width: 50%;}
|
||||
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||||
ul.task-list{list-style: none;}
|
||||
pre > code.sourceCode { white-space: pre; position: relative; }
|
||||
pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
|
||||
pre > code.sourceCode > span:empty { height: 1.2em; }
|
||||
code.sourceCode > span { color: inherit; text-decoration: inherit; }
|
||||
div.sourceCode { margin: 1em 0; }
|
||||
pre.sourceCode { margin: 0; }
|
||||
@media screen {
|
||||
div.sourceCode { overflow: auto; }
|
||||
}
|
||||
@media print {
|
||||
pre > code.sourceCode { white-space: pre-wrap; }
|
||||
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
|
||||
}
|
||||
pre.numberSource code
|
||||
{ counter-reset: source-line 0; }
|
||||
pre.numberSource code > span
|
||||
{ position: relative; left: -4em; counter-increment: source-line; }
|
||||
pre.numberSource code > span > a:first-child::before
|
||||
{ content: counter(source-line);
|
||||
position: relative; left: -1em; text-align: right; vertical-align: baseline;
|
||||
border: none; display: inline-block;
|
||||
-webkit-touch-callout: none; -webkit-user-select: none;
|
||||
-khtml-user-select: none; -moz-user-select: none;
|
||||
-ms-user-select: none; user-select: none;
|
||||
padding: 0 4px; width: 4em;
|
||||
color: #aaaaaa;
|
||||
}
|
||||
pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; }
|
||||
div.sourceCode
|
||||
{ }
|
||||
@media screen {
|
||||
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
|
||||
}
|
||||
code span.al { color: #ff0000; font-weight: bold; } /* Alert */
|
||||
code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
|
||||
code span.at { color: #7d9029; } /* Attribute */
|
||||
code span.bn { color: #40a070; } /* BaseN */
|
||||
code span.bu { } /* BuiltIn */
|
||||
code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
|
||||
code span.ch { color: #4070a0; } /* Char */
|
||||
code span.cn { color: #880000; } /* Constant */
|
||||
code span.co { color: #60a0b0; font-style: italic; } /* Comment */
|
||||
code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
|
||||
code span.do { color: #ba2121; font-style: italic; } /* Documentation */
|
||||
code span.dt { color: #902000; } /* DataType */
|
||||
code span.dv { color: #40a070; } /* DecVal */
|
||||
code span.er { color: #ff0000; font-weight: bold; } /* Error */
|
||||
code span.ex { } /* Extension */
|
||||
code span.fl { color: #40a070; } /* Float */
|
||||
code span.fu { color: #06287e; } /* Function */
|
||||
code span.im { } /* Import */
|
||||
code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
|
||||
code span.kw { color: #007020; font-weight: bold; } /* Keyword */
|
||||
code span.op { color: #666666; } /* Operator */
|
||||
code span.ot { color: #007020; } /* Other */
|
||||
code span.pp { color: #bc7a00; } /* Preprocessor */
|
||||
code span.sc { color: #4070a0; } /* SpecialChar */
|
||||
code span.ss { color: #bb6688; } /* SpecialString */
|
||||
code span.st { color: #4070a0; } /* String */
|
||||
code span.va { color: #19177c; } /* Variable */
|
||||
code span.vs { color: #4070a0; } /* VerbatimString */
|
||||
code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
|
||||
</style>
|
||||
<link rel="stylesheet" href="../tufte.css" />
|
||||
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js" type="text/javascript"></script>
|
||||
<!--[if lt IE 9]>
|
||||
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div style="display:none">
|
||||
\(
|
||||
\newcommand{\NOT}{\neg}
|
||||
\newcommand{\AND}{\wedge}
|
||||
\newcommand{\OR}{\vee}
|
||||
\newcommand{\XOR}{\oplus}
|
||||
\newcommand{\IMP}{\Rightarrow}
|
||||
\newcommand{\IFF}{\Leftrightarrow}
|
||||
\newcommand{\TRUE}{\text{True}\xspace}
|
||||
\newcommand{\FALSE}{\text{False}\xspace}
|
||||
\newcommand{\IN}{\,{\in}\,}
|
||||
\newcommand{\NOTIN}{\,{\notin}\,}
|
||||
\newcommand{\TO}{\rightarrow}
|
||||
\newcommand{\DIV}{\mid}
|
||||
\newcommand{\NDIV}{\nmid}
|
||||
\newcommand{\MOD}[1]{\pmod{#1}}
|
||||
\newcommand{\MODS}[1]{\ (\text{mod}\ #1)}
|
||||
\newcommand{\N}{\mathbb N}
|
||||
\newcommand{\Z}{\mathbb Z}
|
||||
\newcommand{\Q}{\mathbb Q}
|
||||
\newcommand{\R}{\mathbb R}
|
||||
\newcommand{\C}{\mathbb C}
|
||||
\newcommand{\cA}{\mathcal A}
|
||||
\newcommand{\cB}{\mathcal B}
|
||||
\newcommand{\cC}{\mathcal C}
|
||||
\newcommand{\cD}{\mathcal D}
|
||||
\newcommand{\cE}{\mathcal E}
|
||||
\newcommand{\cF}{\mathcal F}
|
||||
\newcommand{\cG}{\mathcal G}
|
||||
\newcommand{\cH}{\mathcal H}
|
||||
\newcommand{\cI}{\mathcal I}
|
||||
\newcommand{\cJ}{\mathcal J}
|
||||
\newcommand{\cL}{\mathcal L}
|
||||
\newcommand{\cK}{\mathcal K}
|
||||
\newcommand{\cN}{\mathcal N}
|
||||
\newcommand{\cO}{\mathcal O}
|
||||
\newcommand{\cP}{\mathcal P}
|
||||
\newcommand{\cQ}{\mathcal Q}
|
||||
\newcommand{\cS}{\mathcal S}
|
||||
\newcommand{\cT}{\mathcal T}
|
||||
\newcommand{\cV}{\mathcal V}
|
||||
\newcommand{\cW}{\mathcal W}
|
||||
\newcommand{\cZ}{\mathcal Z}
|
||||
\newcommand{\emp}{\emptyset}
|
||||
\newcommand{\bs}{\backslash}
|
||||
\newcommand{\floor}[1]{\left \lfloor #1 \right \rfloor}
|
||||
\newcommand{\ceil}[1]{\left \lceil #1 \right \rceil}
|
||||
\newcommand{\abs}[1]{\left | #1 \right |}
|
||||
\newcommand{\xspace}{}
|
||||
\newcommand{\proofheader}[1]{\underline{\textbf{#1}}}
|
||||
\)
|
||||
</div>
|
||||
<header id="title-block-header">
|
||||
<h1 class="title">8.1 An Introduction to Running Time</h1>
|
||||
</header>
|
||||
<section>
|
||||
<p>So far in this course, when we have studied programming concepts, we have focused on the <em>correctness</em> of our code. In Chapters 1–5, we learned about different programming constructs, understanding what to do, how to combine them into larger programs, and how to test these programs to make sure they are correct. In Chapters 6 and 7, we learned about mathematical proof, and applied this skill to proving the correctness of various algorithms, including every part of the RSA cryptosystem.</p>
|
||||
<p>Yet when it comes to evaluating programs, correctness is not the only important measure. As we alluded to in Chapter 7, the amount of time a program takes to run, or program <em>running time</em>, is a critical consideration.<label for="sn-0" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-0" class="margin-toggle"/><span class="sidenote"> Running time is often shortened to “runtime”, and is also know as the “efficiency” or “performance” of a program.</span> In this chapter, we’ll study a formal approach to analysing the running time of a program. This section will introduce the topic, and then in future sections we’ll build up some mathematical theory about comparing rates of function growth, and then apply this theory to real program code.</p>
|
||||
<h2 id="how-do-we-measure-running-time">How do we measure running time?</h2>
|
||||
<p>Consider the following function, which prints out the first <code>n</code> natural numbers:</p>
|
||||
<div class="sourceCode" id="cb1"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1"></a><span class="kw">def</span> print_integers(n: <span class="bu">int</span>) <span class="op">-></span> <span class="va">None</span>:</span>
|
||||
<span id="cb1-2"><a href="#cb1-2"></a> <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, n):</span>
|
||||
<span id="cb1-3"><a href="#cb1-3"></a> <span class="bu">print</span>(i)</span></code></pre></div>
|
||||
<p>What can we say about the running time of this function? An empirical approach would be to measure the time it takes for this function to run on a bunch of different inputs, and then take the average of these times to come up with some sort of estimate of the “average” running time.</p>
|
||||
<p>But of course, given that this algorithm performs an action for every natural number between <code>0</code> and <code>n - 1</code>, we expect it to take longer as <code>n</code> gets larger, so taking an average of a bunch of running times loses important information about the inputs.<label for="sn-1" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-1" class="margin-toggle"/><span class="sidenote">This is like doing a random poll of how many birthday cakes people have eaten without taking into account how old the respondents are.</span></p>
|
||||
<p>How about choosing one particular input, calling the function multiple times on that input, and averaging those running times? This seems better, but even here there are some problems. For one, the computer’s hardware can affect running time; for another, computers all are running multiple programs at the same time, so what else is currently running on your computer also affects running time. So even running this experiment on one computer wouldn’t necessarily be indicative of how long the function would take on a different computer, nor even how long it would take on the same computer running a different number of other programs.</p>
|
||||
<p>While these sorts of timing experiments are actually done in practice for evaluating particular hardware or extremely low-level (close to hardware) programs, these details are often not helpful for most software developers, as they do not have control over the machine on which their software will be run.<label for="sn-2" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-2" class="margin-toggle"/><span class="sidenote"> That said, these timing experiments can provide an intuitive understanding of the efficiency of our programs. We will explore how to conduct basic timing experiments at the end of this chapter.</span></p>
|
||||
<p>So rather than use an empirical measurement of runtime, what we do instead is use an abstract representation of runtime: the number of “basic operations” an algorithm executes. This means that we can analyze functions without needing a computer, and our analysis theoretically applies to any computer system. However, there is a good reason “basic operation” is in quotation marks—this vague term raises a whole slew of questions:</p>
|
||||
<ul>
|
||||
<li>What counts as a “basic operation”?</li>
|
||||
<li>How do we tell which “basic operations” are used by an algorithm?</li>
|
||||
<li>Do all “basic operations” take the same amount of time?</li>
|
||||
</ul>
|
||||
<p>The answers to these questions can depend on the hardware being used, as well as what programming language the algorithm is written in. Of course, these are precisely the details we wish to avoid thinking about. In this section, we will count only the calls to <code>print</code> as basic operations, and study <code>print_integers</code> and some variations to establish some intuition and terminology.</p>
|
||||
<h2 id="linear-running-time">Linear running time</h2>
|
||||
<p>First, let’s return to <code>print_integers</code>.</p>
|
||||
<div class="sourceCode" id="cb2"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1"></a><span class="kw">def</span> print_integers(n: <span class="bu">int</span>) <span class="op">-></span> <span class="va">None</span>:</span>
|
||||
<span id="cb2-2"><a href="#cb2-2"></a> <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, n):</span>
|
||||
<span id="cb2-3"><a href="#cb2-3"></a> <span class="bu">print</span>(i)</span></code></pre></div>
|
||||
<p>From Chapter 4, we know that the for loop will call <code>print</code> once per iteration. We also know that this loop iterates <span class="math inline">\(n\)</span> times (with <code>i</code> taking on the values 0, 1, 2, …, <span class="math inline">\(n - 1\)</span>):</p>
|
||||
<div class="sourceCode" id="cb3"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1"></a><span class="op">>>></span> print_integers(<span class="dv">2</span>)</span>
|
||||
<span id="cb3-2"><a href="#cb3-2"></a><span class="dv">0</span></span>
|
||||
<span id="cb3-3"><a href="#cb3-3"></a><span class="dv">1</span></span>
|
||||
<span id="cb3-4"><a href="#cb3-4"></a><span class="op">>>></span> print_integers(<span class="dv">4</span>)</span>
|
||||
<span id="cb3-5"><a href="#cb3-5"></a><span class="dv">0</span></span>
|
||||
<span id="cb3-6"><a href="#cb3-6"></a><span class="dv">1</span></span>
|
||||
<span id="cb3-7"><a href="#cb3-7"></a><span class="dv">2</span></span>
|
||||
<span id="cb3-8"><a href="#cb3-8"></a><span class="dv">3</span></span>
|
||||
<span id="cb3-9"><a href="#cb3-9"></a><span class="op">>>></span> print_integers(<span class="dv">8</span>)</span>
|
||||
<span id="cb3-10"><a href="#cb3-10"></a><span class="dv">0</span></span>
|
||||
<span id="cb3-11"><a href="#cb3-11"></a><span class="dv">1</span></span>
|
||||
<span id="cb3-12"><a href="#cb3-12"></a><span class="dv">2</span></span>
|
||||
<span id="cb3-13"><a href="#cb3-13"></a><span class="dv">3</span></span>
|
||||
<span id="cb3-14"><a href="#cb3-14"></a><span class="dv">4</span></span>
|
||||
<span id="cb3-15"><a href="#cb3-15"></a><span class="dv">5</span></span>
|
||||
<span id="cb3-16"><a href="#cb3-16"></a><span class="dv">6</span></span>
|
||||
<span id="cb3-17"><a href="#cb3-17"></a><span class="dv">7</span></span></code></pre></div>
|
||||
<p>So then for an input <span class="math inline">\(n\)</span>, there are <span class="math inline">\(n\)</span> calls to <code>print</code>. We say that the running time of <code>print_integers</code> on input <span class="math inline">\(n\)</span> is <span class="math inline">\(n\)</span> basic operations. If we plot <span class="math inline">\(n\)</span> against this measure running time, we obtain a line:</p>
|
||||
<p>We say that <code>print_integers</code> has a <strong>linear</strong> running time, as the number of basic operations is a linear function of the input <span class="math inline">\(n\)</span>.</p>
|
||||
<p><img src="images/Linear.png" /></p>
|
||||
<h2 id="quadratic-running-time">Quadratic running time</h2>
|
||||
<p>Let us now consider a function that prints all combinations of pairs of integers:</p>
|
||||
<div class="sourceCode" id="cb4"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb4-1"><a href="#cb4-1"></a><span class="kw">def</span> print_pairs(n: <span class="bu">int</span>) <span class="op">-></span> <span class="va">None</span>:</span>
|
||||
<span id="cb4-2"><a href="#cb4-2"></a> <span class="co">"""Print all combinations of pairs of the first n natural numbers."""</span></span>
|
||||
<span id="cb4-3"><a href="#cb4-3"></a> <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, n):</span>
|
||||
<span id="cb4-4"><a href="#cb4-4"></a> <span class="cf">for</span> j <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, n):</span>
|
||||
<span id="cb4-5"><a href="#cb4-5"></a> <span class="bu">print</span>(i, j)</span></code></pre></div>
|
||||
<p>What is the running time of this function? Similar to our previous example, there is a for loop that calls <code>print</code> <span class="math inline">\(n\)</span> times, but now this loop is nested inside another for loop. Let’s see some examples of this function being called:</p>
|
||||
<div class="sourceCode" id="cb5"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb5-1"><a href="#cb5-1"></a><span class="op">>>></span> print_pairs(<span class="dv">1</span>)</span>
|
||||
<span id="cb5-2"><a href="#cb5-2"></a><span class="dv">0</span> <span class="dv">0</span></span>
|
||||
<span id="cb5-3"><a href="#cb5-3"></a><span class="op">>>></span> print_pairs(<span class="dv">2</span>)</span>
|
||||
<span id="cb5-4"><a href="#cb5-4"></a><span class="dv">0</span> <span class="dv">0</span></span>
|
||||
<span id="cb5-5"><a href="#cb5-5"></a><span class="dv">0</span> <span class="dv">1</span></span>
|
||||
<span id="cb5-6"><a href="#cb5-6"></a><span class="dv">1</span> <span class="dv">0</span></span>
|
||||
<span id="cb5-7"><a href="#cb5-7"></a><span class="dv">1</span> <span class="dv">1</span></span>
|
||||
<span id="cb5-8"><a href="#cb5-8"></a><span class="op">>>></span> print_pairs(<span class="dv">3</span>)</span>
|
||||
<span id="cb5-9"><a href="#cb5-9"></a><span class="dv">0</span> <span class="dv">0</span></span>
|
||||
<span id="cb5-10"><a href="#cb5-10"></a><span class="dv">0</span> <span class="dv">1</span></span>
|
||||
<span id="cb5-11"><a href="#cb5-11"></a><span class="dv">0</span> <span class="dv">2</span></span>
|
||||
<span id="cb5-12"><a href="#cb5-12"></a><span class="dv">1</span> <span class="dv">0</span></span>
|
||||
<span id="cb5-13"><a href="#cb5-13"></a><span class="dv">1</span> <span class="dv">1</span></span>
|
||||
<span id="cb5-14"><a href="#cb5-14"></a><span class="dv">1</span> <span class="dv">2</span></span>
|
||||
<span id="cb5-15"><a href="#cb5-15"></a><span class="dv">2</span> <span class="dv">0</span></span>
|
||||
<span id="cb5-16"><a href="#cb5-16"></a><span class="dv">2</span> <span class="dv">1</span></span>
|
||||
<span id="cb5-17"><a href="#cb5-17"></a><span class="dv">2</span> <span class="dv">2</span></span></code></pre></div>
|
||||
<p>If we look at the outer loop (loop variable <code>i</code>), we see that it repeats its body <span class="math inline">\(n\)</span> times. But its body is another loop, which repeats <em>its</em> body <span class="math inline">\(n\)</span> times. So the inner loop takes <span class="math inline">\(n\)</span> calls to print each time it is executed, and it is executed <span class="math inline">\(n\)</span> times in total. This means <code>print</code> is called <span class="math inline">\(n^2\)</span> times in total.</p>
|
||||
<p>We say that <code>print_pairs</code> has a <strong>quadratic</strong> running time, as the number of basic operations is a quadratic function of the input <span class="math inline">\(n\)</span>.</p>
|
||||
<p><img src="images/Quadratic.png" /></p>
|
||||
<h2 id="logarithmic-running-time">Logarithmic running time</h2>
|
||||
<p>Now let’s consider the following function, which prints out the powers of two that are less than a positive integer <span class="math inline">\(n\)</span>.<label for="sn-3" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-3" class="margin-toggle"/><span class="sidenote"> These numbers are of the form <span class="math inline">\(2^i\)</span>, where <span class="math inline">\(i\)</span> can range from 0 to <span class="math inline">\(\ceil{\log_2(n)} - 1\)</span>. For example, when <span class="math inline">\(n = 16\)</span>, <span class="math inline">\(\ceil{\log_2(n)} = 4\)</span>, and <span class="math inline">\(i\)</span> ranges from 0 to 3. When <span class="math inline">\(n = 7\)</span>, <span class="math inline">\(\ceil{\log_2(n)} = 3\)</span>, and <span class="math inline">\(i\)</span> ranges from 0 to 2.</span></p>
|
||||
<div class="sourceCode" id="cb6"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb6-1"><a href="#cb6-1"></a><span class="kw">def</span> print_powers_of_two(n: <span class="bu">int</span>) <span class="op">-></span> <span class="va">None</span>:</span>
|
||||
<span id="cb6-2"><a href="#cb6-2"></a> <span class="co">"""Print the powers of two that are less than n.</span></span>
|
||||
<span id="cb6-3"><a href="#cb6-3"></a></span>
|
||||
<span id="cb6-4"><a href="#cb6-4"></a><span class="co"> Preconditions:</span></span>
|
||||
<span id="cb6-5"><a href="#cb6-5"></a><span class="co"> - n > 0</span></span>
|
||||
<span id="cb6-6"><a href="#cb6-6"></a><span class="co"> """</span></span>
|
||||
<span id="cb6-7"><a href="#cb6-7"></a> <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, math.ceil(math.log2(n))):</span>
|
||||
<span id="cb6-8"><a href="#cb6-8"></a> <span class="bu">print</span>(<span class="dv">2</span> <span class="op">**</span> i)</span></code></pre></div>
|
||||
<p>In this case, the number of calls to <code>print</code> is <span class="math inline">\(\ceil{\log_2(n)}\)</span>. So the running time of <code>print_powers_of_two</code> is <em>approximately</em>, but not exactly, <span class="math inline">\(\log_2(n)\)</span>. Yet in this case we still say that <code>print_powers_of_two</code> has a <strong>logarithmic</strong> running time.</p>
|
||||
<p><img src="images/Logarithmic.png" /></p>
|
||||
<h2 id="constant-running-time">Constant running time</h2>
|
||||
<p>Our final example in this section is a bit unusual.</p>
|
||||
<div class="sourceCode" id="cb7"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb7-1"><a href="#cb7-1"></a><span class="kw">def</span> print_ten(n: <span class="bu">int</span>) <span class="op">-></span> <span class="va">None</span>:</span>
|
||||
<span id="cb7-2"><a href="#cb7-2"></a> <span class="co">"""Print n ten times."""</span></span>
|
||||
<span id="cb7-3"><a href="#cb7-3"></a> <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, <span class="dv">10</span>):</span>
|
||||
<span id="cb7-4"><a href="#cb7-4"></a> <span class="bu">print</span>(n)</span></code></pre></div>
|
||||
<p>How many times is <code>print</code> called here? We can again tell from the header of the for loop: this loop iterates 10 times, and so <code>print</code> is called 10 times, <em>regardless of what <span class="math inline">\(n\)</span> is</em>!</p>
|
||||
<p>We say that <code>print_ten</code> has a <strong>constant</strong> running time, as the number of basic operations is independent to the input size.</p>
|
||||
<p><img src="images/Constant.png" /></p>
|
||||
<h2 id="basic-operations">Basic operations</h2>
|
||||
<p>In the past four examples, we have seen examples of functions that have linear, quadratic, logarithmic, and constant running times. While these labels are not precise, they do give us intuition about the relative size of each function.</p>
|
||||
<!--  -->
|
||||
<p>Functions with linear running time are faster than ones with quadratic running time, and slower than ones with logarithmic running time. Functions with a constant running time are the fastest of all.</p>
|
||||
<p>But all of our informal analyses in the previous section relied on defining a “basic operation” to be a call to <code>print</code>. We said, for example, that the running time of <code>print_integers</code> had a running time of <span class="math inline">\(n\)</span>. But what if we had a friend comes along and say, “No wait, the variable <code>i</code> must be assigned a new value at every loop iteration, and that counts as a basic operation.” Okay, so then we would say that there are <span class="math inline">\(n\)</span> <code>print</code> calls and <span class="math inline">\(n\)</span> assignments to <code>i</code>, for a total running time of <span class="math inline">\(2n\)</span> basic operations for an input <span class="math inline">\(n\)</span>.</p>
|
||||
<p>But then another friend chimes in, saying “But <code>print</code> calls take longer than variable assignments, since they need to change pixels on your monitor, so you should count each <code>print</code> call as <span class="math inline">\(10\)</span> basic operations.” Okay, so then there are <span class="math inline">\(n\)</span> print calls worth <span class="math inline">\(10n\)</span> basic operations, plus the assignments to <code>i</code>, for a total of <span class="math inline">\(11n\)</span> basic operations for an input <span class="math inline">\(n\)</span>.</p>
|
||||
<p>And then another friend joins in: “But you need to factor in an overhead of calling the function as a first step before the body executes, which counts as <span class="math inline">\(1.5\)</span> basic operations (slower than assignment, faster than <code>print</code>).” So then we now have a running time of <span class="math inline">\(11n + 1.5\)</span> basic operations for an input <span class="math inline">\(n\)</span>.</p>
|
||||
<p>And then another friend starts to speak, but you cut them off and say “That’s it! This is getting way too complicated. I’m going back to timing experiments, which may be inaccurate but at least I won’t have to listen to these increasing levels of fussiness.”</p>
|
||||
<p>The expressions <span class="math inline">\(n\)</span>, <span class="math inline">\(2n\)</span>, <span class="math inline">\(11n\)</span>, and <span class="math inline">\(11n + 1.5\)</span> may be different mathematically, but they share a common qualitative type of growth: they are all <em>linear</em>. And so we know, at least intuitively, that they are all faster than quadratic running times and slower than logarithmic running times. What we will study in the next section is how to make this observation precise, and thus avoid the tedium of trying to exactly quantify our “basic operations”, and instead measure the overall rate of growth in the number of operations.</p>
|
||||
<h2 id="references">References</h2>
|
||||
<ul>
|
||||
<li>CSC108 videos: <a href="https://www.youtube.com/watch?v=2QgtAWzBVuk&list=PLfMGJf6SEIv7uysvvsaoknnzfrCCMMk5r">Algorithm Analysis</a></li>
|
||||
</ul>
|
||||
</section>
|
||||
<!--
|
||||
Let us now consider a very similar function, `print_odd_integers`:
|
||||
|
||||
```python
|
||||
def print_odd_integers(n: int) -> None:
|
||||
"""Print the odd integers from 1 to n, inclusive.
|
||||
"""
|
||||
for i in range(1, n + 1, 2):
|
||||
print(i)
|
||||
```
|
||||
|
||||
As the name implies, this function will only print odd integers up to n:
|
||||
|
||||
```python
|
||||
>>> print_odd_integers(2)
|
||||
1
|
||||
>>> print_odd_integers(4)
|
||||
1
|
||||
3
|
||||
>>> print_odd_integers(8)
|
||||
1
|
||||
3
|
||||
5
|
||||
7
|
||||
```
|
||||
|
||||
For `print_integers`, the loop repeats $n$ times.
|
||||
But for `print_odd_integers`, the loop repeats roughly $n / 2$ times.
|
||||
Despite this difference, we say that both functions have **linear** runtime scaling.
|
||||
That is, while the slopes of the lines may be different, the lines themselves still grow proportionally with the size of $n$. -->
|
||||
<footer>
|
||||
<a href="https://www.teach.cs.toronto.edu/~csc110y/fall/notes/">CSC110 Course Notes Home</a>
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,213 @@
|
||||
<!DOCTYPE html>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="generator" content="pandoc" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||||
<title>8.2 Comparing Asymptotic Function Growth with Big-O</title>
|
||||
<style>
|
||||
code{white-space: pre-wrap;}
|
||||
span.smallcaps{font-variant: small-caps;}
|
||||
span.underline{text-decoration: underline;}
|
||||
div.column{display: inline-block; vertical-align: top; width: 50%;}
|
||||
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||||
ul.task-list{list-style: none;}
|
||||
</style>
|
||||
<link rel="stylesheet" href="../tufte.css" />
|
||||
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js" type="text/javascript"></script>
|
||||
<!--[if lt IE 9]>
|
||||
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div style="display:none">
|
||||
\(
|
||||
\newcommand{\NOT}{\neg}
|
||||
\newcommand{\AND}{\wedge}
|
||||
\newcommand{\OR}{\vee}
|
||||
\newcommand{\XOR}{\oplus}
|
||||
\newcommand{\IMP}{\Rightarrow}
|
||||
\newcommand{\IFF}{\Leftrightarrow}
|
||||
\newcommand{\TRUE}{\text{True}\xspace}
|
||||
\newcommand{\FALSE}{\text{False}\xspace}
|
||||
\newcommand{\IN}{\,{\in}\,}
|
||||
\newcommand{\NOTIN}{\,{\notin}\,}
|
||||
\newcommand{\TO}{\rightarrow}
|
||||
\newcommand{\DIV}{\mid}
|
||||
\newcommand{\NDIV}{\nmid}
|
||||
\newcommand{\MOD}[1]{\pmod{#1}}
|
||||
\newcommand{\MODS}[1]{\ (\text{mod}\ #1)}
|
||||
\newcommand{\N}{\mathbb N}
|
||||
\newcommand{\Z}{\mathbb Z}
|
||||
\newcommand{\Q}{\mathbb Q}
|
||||
\newcommand{\R}{\mathbb R}
|
||||
\newcommand{\C}{\mathbb C}
|
||||
\newcommand{\cA}{\mathcal A}
|
||||
\newcommand{\cB}{\mathcal B}
|
||||
\newcommand{\cC}{\mathcal C}
|
||||
\newcommand{\cD}{\mathcal D}
|
||||
\newcommand{\cE}{\mathcal E}
|
||||
\newcommand{\cF}{\mathcal F}
|
||||
\newcommand{\cG}{\mathcal G}
|
||||
\newcommand{\cH}{\mathcal H}
|
||||
\newcommand{\cI}{\mathcal I}
|
||||
\newcommand{\cJ}{\mathcal J}
|
||||
\newcommand{\cL}{\mathcal L}
|
||||
\newcommand{\cK}{\mathcal K}
|
||||
\newcommand{\cN}{\mathcal N}
|
||||
\newcommand{\cO}{\mathcal O}
|
||||
\newcommand{\cP}{\mathcal P}
|
||||
\newcommand{\cQ}{\mathcal Q}
|
||||
\newcommand{\cS}{\mathcal S}
|
||||
\newcommand{\cT}{\mathcal T}
|
||||
\newcommand{\cV}{\mathcal V}
|
||||
\newcommand{\cW}{\mathcal W}
|
||||
\newcommand{\cZ}{\mathcal Z}
|
||||
\newcommand{\emp}{\emptyset}
|
||||
\newcommand{\bs}{\backslash}
|
||||
\newcommand{\floor}[1]{\left \lfloor #1 \right \rfloor}
|
||||
\newcommand{\ceil}[1]{\left \lceil #1 \right \rceil}
|
||||
\newcommand{\abs}[1]{\left | #1 \right |}
|
||||
\newcommand{\xspace}{}
|
||||
\newcommand{\proofheader}[1]{\underline{\textbf{#1}}}
|
||||
\)
|
||||
</div>
|
||||
<header id="title-block-header">
|
||||
<h1 class="title">8.2 Comparing Asymptotic Function Growth with Big-O</h1>
|
||||
</header>
|
||||
<section>
|
||||
<p>In the previous section, we began our study of program running time with a few simple examples to guide our intuition. One question that emerged from these examples was how we define what “basic operations” we actually count when analysing a program’s running time—or better yet, how we can ignore small differences in counts that result from slighly different definitions of “basic operation”. This question grows even more important as we study more complex algorithms consisting of many lines of code.</p>
|
||||
<p>Over the next two sections, we’ll develop a powerful mathematical tool for comparing function growth rates. This will formalize the idea of “linear”, “quadratic”, “logarithmic”, and “constant” running times from the previous section, and extend these categories to all types of functions.</p>
|
||||
<h2 id="four-kinds-of-dominance">Four kinds of dominance</h2>
|
||||
<p>Here is a quick reminder about function notation. When we write <span class="math inline">\(f : A \to B\)</span>, we say that <span class="math inline">\(f\)</span> is a function which maps elements of <span class="math inline">\(A\)</span> to elements of <span class="math inline">\(B\)</span>. In this chapter, we will mainly be concerned about functions mapping the natural numbers to the nonnegative real numbers,<label for="sn-0" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-0" class="margin-toggle"/><span class="sidenote">These are the domain and codomain which arise in algorithm analysis—an algorithm can’t take “negative” time to run, after all.</span> i.e., functions <span class="math inline">\(f: \N \to \R^{\geq 0}\)</span>. Though there are many different properties of functions that mathematicians study, we are only going to look at one such property: describing the long-term (i.e., <strong>asymptotic</strong>) growth of a function. We will proceed by building up a few different definitions of comparing function growth, which will eventually lead into one which is robust enough to be used in practice.</p>
|
||||
<div class="definition" data-terms="dominates (absolute)">
|
||||
<p>Let <span class="math inline">\(f, g : \N \to \R^{\ge 0}\)</span>. We say that <span class="math inline">\(g\)</span> is <strong>absolutely dominated by</strong> <span class="math inline">\(f\)</span> when for all <span class="math inline">\(n \in \N\)</span>, <span class="math inline">\(g(n) \leq f(n)\)</span>.</p>
|
||||
</div>
|
||||
<div class="example">
|
||||
<p>Let <span class="math inline">\(f(n) = n^2\)</span> and <span class="math inline">\(g(n) = n\)</span>. Prove that <span class="math inline">\(g\)</span> is absolutely dominated by <span class="math inline">\(f\)</span>.</p>
|
||||
<div class="translation">
|
||||
<p>This is a straightforward unpacking of a definition, which you should be very comfortable with by now: <span class="math inline">\(\forall n \in \N,~ g(n) \leq f(n)\)</span>.<label for="sn-1" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-1" class="margin-toggle"/><span class="sidenote">Note that we aren’t quantifying over <span class="math inline">\(f\)</span> and <span class="math inline">\(g\)</span>; the “let” in the example defines concrete functions that we want to prove something about.</span></p>
|
||||
</div>
|
||||
<div class="proof">
|
||||
<p>Let <span class="math inline">\(n \in \N\)</span>. We want to show that <span class="math inline">\(n \leq n^2\)</span>.</p>
|
||||
<p><strong>Case 1</strong>: Assume <span class="math inline">\(n = 0\)</span>. In this case, <span class="math inline">\(n^2 = n = 0\)</span>, so the inequality holds.</p>
|
||||
<p><strong>Case 2</strong>: Assume <span class="math inline">\(n \geq 1\)</span>. In this case, we take the inequality <span class="math inline">\(n \geq 1\)</span> and multiply both sides by <span class="math inline">\(n\)</span> to get <span class="math inline">\(n^2 \geq n\)</span>, or equivalently <span class="math inline">\(n \leq n^2.\)</span></p>
|
||||
</div>
|
||||
</div>
|
||||
<p>Unfortunately, absolute dominance is too strict for our purposes: if <span class="math inline">\(g(n) \leq f(n)\)</span> for every natural number except <span class="math inline">\(5\)</span>, then we can’t say that <span class="math inline">\(g\)</span> is absolutely dominated by <span class="math inline">\(f\)</span>. For example, the function <span class="math inline">\(g(n) = 2n\)</span> is not absolutely dominated by <span class="math inline">\(f(n) = n^2\)</span>, even though <span class="math inline">\(g(n) \leq f(n)\)</span> everywhere except <span class="math inline">\(n = 1\)</span>. Graphically:</p>
|
||||
<p><img src="images/LinearQuadratic.png" alt="Linear vs. Quadratic Runtime" /><br />
|
||||
</p>
|
||||
<p>Here is another definition which is a bit more flexible than absolute dominance.</p>
|
||||
<div class="definition" data-terms="dominates (up to a constant factor)">
|
||||
<p>Let <span class="math inline">\(f, g : \N \to \R^{\ge 0}\)</span>. We say that <span class="math inline">\(g\)</span> <strong>is dominated by <span class="math inline">\(f\)</span> up to a constant factor</strong> when there exists a positive real number <span class="math inline">\(c\)</span> such that for all <span class="math inline">\(n \in \N\)</span>, <span class="math inline">\(g(n) \leq c \cdot f(n)\)</span>.</p>
|
||||
</div>
|
||||
<div class="example">
|
||||
<p>Let <span class="math inline">\(f(n) = n^2\)</span> and <span class="math inline">\(g(n) = 2n\)</span>. Prove that <span class="math inline">\(g\)</span> is dominated by <span class="math inline">\(f\)</span> up to a constant factor.</p>
|
||||
<div class="translation">
|
||||
<p>Once again, the translation is a simple unpacking of the previous definition:<label for="sn-2" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-2" class="margin-toggle"/><span class="sidenote">Remember: the order of quantifiers matters! The choice of <span class="math inline">\(c\)</span> is <em>not</em> allowed to depend on <span class="math inline">\(n\)</span>.</span></p>
|
||||
<p><span class="math display">\[\exists c \in \R^+,~ \forall n \in \N,~ g(n) \leq c \cdot f(n).\]</span></p>
|
||||
</div>
|
||||
<div class="discussion">
|
||||
<p>The term “constant factor” is revealing. We already saw that <span class="math inline">\(n\)</span> is absolutely dominated by <span class="math inline">\(n^2\)</span>, so if the <span class="math inline">\(n\)</span> is multiplied by <span class="math inline">\(2\)</span>, then we should be able to multiply <span class="math inline">\(n^2\)</span> by <span class="math inline">\(2\)</span> as well to get the calculation to work out.</p>
|
||||
</div>
|
||||
<div class="proof">
|
||||
<p>Let <span class="math inline">\(c = 2\)</span>, and let <span class="math inline">\(n \in \N\)</span>. We want to prove that <span class="math inline">\(g(n) \leq c \cdot f(n)\)</span>, or in other words, <span class="math inline">\(2n \leq 2n^2\)</span>.</p>
|
||||
<p><strong>Case 1</strong>: Assume <span class="math inline">\(n = 0\)</span>. In this case, <span class="math inline">\(2n = 0\)</span> and <span class="math inline">\(2n^2 = 0\)</span>, so the inequality holds.</p>
|
||||
<p><strong>Case 2</strong>: Assume <span class="math inline">\(n \geq 1\)</span>. Taking the assumed inequality <span class="math inline">\(n \geq 1\)</span> and multiplying both sides by <span class="math inline">\(2n\)</span> yields <span class="math inline">\(2n^2 \geq 2n\)</span>, or equivalently <span class="math inline">\(2n \leq 2n^2\)</span>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p>Intuitively, “dominated by up to a constant factor” allows us to ignore multiplicative constants in our functions. This will be very useful in our running time analysis because it frees us from worrying about the exact constants used to represent numbers of basic operations: <span class="math inline">\(n\)</span>, <span class="math inline">\(2n\)</span>, and <span class="math inline">\(11n\)</span> are all <em>equivalent</em> in the sense that each one dominates the other two up to a constant factor.</p>
|
||||
<p>However, this second definition is still a little too restrictive, as the inequality must hold for every value of <span class="math inline">\(n\)</span>. Consider the functions <span class="math inline">\(f(n) = n^2\)</span> and <span class="math inline">\(g(n) = n + 90\)</span>. No matter how much we scale up <span class="math inline">\(f\)</span> by multiplying it by a constant, <span class="math inline">\(f(0)\)</span> will always be less than <span class="math inline">\(g(0)\)</span>, so we cannot say that <span class="math inline">\(g\)</span> is dominated by <span class="math inline">\(f\)</span> up to a constant factor. And again this is silly: it is certainly possible to find a constant <span class="math inline">\(c\)</span> such that <span class="math inline">\(g(n) \leq cf(n)\)</span> for every value except <span class="math inline">\(n = 0\)</span>. So we want some way of omitting the value <span class="math inline">\(n = 0\)</span> from consideration; this is precisely what our third definition gives us.</p>
|
||||
<div class="definition" data-terms="dominates (eventually)">
|
||||
<p>Let <span class="math inline">\(f, g : \N \to \R^{\ge 0}\)</span>. We say that <span class="math inline">\(g\)</span> <strong>is eventually dominated by</strong> <span class="math inline">\(f\)</span> when there exists <span class="math inline">\(n_0 \in \R^+\)</span> such that <span class="math inline">\(\forall n \in \N\)</span>, if <span class="math inline">\(n \geq n_0\)</span> then <span class="math inline">\(g(n) \leq f(n)\)</span>.</p>
|
||||
</div>
|
||||
<div class="example">
|
||||
<p>Let <span class="math inline">\(f(n) = n^2\)</span> and <span class="math inline">\(g(n) = n + 90\)</span>. Prove that <span class="math inline">\(g\)</span> is eventually dominated by <span class="math inline">\(f\)</span>.</p>
|
||||
<div class="translation">
|
||||
<p><span class="math display">\[\exists n_0 \in \R^+,~ \forall n \in \N,~ n \geq n_0 \IMP g(n) \leq f(n).\]</span></p>
|
||||
</div>
|
||||
<div class="discussion">
|
||||
<p>Okay, so rather than finding a constant to scale up <span class="math inline">\(f\)</span>, we need to argue that for “large enough” values of <span class="math inline">\(n\)</span>, <span class="math inline">\(n + 90 \leq n^2\)</span>. How do we know that value of <span class="math inline">\(n\)</span> is “large enough?”</p>
|
||||
<p>Since this is a quadratic inequality, it is actually possible to solve it directly using factoring or the quadratic formula. But that’s not really the point of this example, so instead we’ll take advantage of the fact that <em>we</em> get to choose the value of <span class="math inline">\(n_0\)</span> to pick one which is large enough.</p>
|
||||
</div>
|
||||
<div class="proof">
|
||||
<p>Let <span class="math inline">\(n_0 = 90\)</span>, let <span class="math inline">\(n \in \N\)</span>, and assume <span class="math inline">\(n \geq n_0\)</span>. We want to prove that <span class="math inline">\(n + 90 \leq n^2\)</span>.</p>
|
||||
<p>We will start with the left-hand side and obtain a chain of inequalities that lead to the right-hand side. <span class="math display">\[\begin{align*}
|
||||
n + 90 &\leq n + n \tag{since $n \geq 90$} \\
|
||||
&= 2n \\
|
||||
&\leq n \cdot n \tag{since $n \geq 2$} \\
|
||||
&= n^2
|
||||
\end{align*}\]</span></p>
|
||||
</div>
|
||||
</div>
|
||||
<p>Intuitively, this definition allows us to ignore “small” values of <span class="math inline">\(n\)</span> and focus on the long term, or asymptotic, behaviour of the function. This is particularly important for ignoring the influence of slow-growing terms in a function, which may affect the function values for “small” <span class="math inline">\(n\)</span>, but eventually are overshadowed by the faster-growing terms. In the above example, we knew that <span class="math inline">\(n^2\)</span> grows faster than <span class="math inline">\(n\)</span>, but because an extra <span class="math inline">\(+ 90\)</span> was added to the latter function, it took a while for the faster growth rate of <span class="math inline">\(n^2\)</span> to “catch up” to <span class="math inline">\(n + 90\)</span>.</p>
|
||||
<p>Our final definition combines both of the previous ones, enabling us to ignore both <em>constant factors</em> and <em>small values of <span class="math inline">\(n\)</span></em> when comparing functions.</p>
|
||||
<div class="definition" data-terms="Big-O">
|
||||
<p>Let <span class="math inline">\(f, g: \N \to \R^{\ge 0}\)</span>. We say that <span class="math inline">\(g\)</span> <strong>is eventually dominated by <span class="math inline">\(f\)</span> up to a constant factor</strong> when there exist <span class="math inline">\(c, n_0 \in \R^+\)</span>, such that for all <span class="math inline">\(n \in \N\)</span>, if <span class="math inline">\(n \geq n_0\)</span> then <span class="math inline">\(g(n) \leq c \cdot f(n)\)</span>.</p>
|
||||
<p>In this case, we also say that <strong><span class="math inline">\(g\)</span> is Big-O of <span class="math inline">\(f\)</span></strong>, and write <span class="math inline">\(g \in \cO(f)\)</span>.</p>
|
||||
<p>We use the notation “<span class="math inline">\(\in \cO(f)\)</span>” here because we formally define <span class="math inline">\(\cO(f)\)</span> to be the <em>set</em> of functions that are eventually dominated by <span class="math inline">\(f\)</span> up to a constant factor: <span class="math display">\[\cO(f) = \{g \mid g: \N \to \R^{\ge 0},~\text{and}~\exists c, n_0 \in \R^+,~ \forall n \in \N,~ n \geq n_0 \IMP g(n) \leq c \cdot f(n)\}.\]</span></p>
|
||||
</div>
|
||||
<div class="example">
|
||||
<p>Let <span class="math inline">\(f(n) = n^3\)</span> and <span class="math inline">\(g(n) = n^3 + 100n + 5000\)</span>. Prove that <span class="math inline">\(g \in \cO(f)\)</span>.<label for="sn-3" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-3" class="margin-toggle"/><span class="sidenote">We can also express this statement as “<span class="math inline">\(n^3 + 100n + 5000 \in \cO(n^3)\)</span>”.</span></p>
|
||||
<div class="translation">
|
||||
<p><span class="math display">\[\exists c, n_0 \in \R^+,~ \forall n \in \N,~ n \geq n_0 \IMP n^3 + 100n + 5000 \leq c n^3.\]</span></p>
|
||||
</div>
|
||||
<div class="discussion">
|
||||
<p>It’s worth pointing out that in this case, <span class="math inline">\(g\)</span> is neither eventually dominated by <span class="math inline">\(f\)</span> nor dominated by <span class="math inline">\(f\)</span> up to a constant factor.<label for="sn-4" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-4" class="margin-toggle"/><span class="sidenote"> Exercise: prove this!</span> So we’ll really need to make use of both constants <span class="math inline">\(c\)</span> and <span class="math inline">\(n_0\)</span>. They’re both existentially-quantified, so we have a lot of freedom in how to choose them!</p>
|
||||
<p>Here’s an idea: let’s split up the inequality <span class="math inline">\(n^3 + 100n + 5000 \leq c n^3\)</span> into three simpler ones: <span class="math display">\[\begin{align*}
|
||||
n^3 &\leq c_1 n^3 \\
|
||||
100n &\leq c_2 n^3 \\
|
||||
5000 &\leq c_3 n^3
|
||||
\end{align*}\]</span></p>
|
||||
<p>If we can make these three inequalities true, adding them together will give us our desired result (setting <span class="math inline">\(c = c_1 + c_2 + c_3\)</span>). Each of these inequalities is simple enough that we can “solve” them by inspection. Moreover, because we have freedom in how we choose <span class="math inline">\(n_0\)</span> and <span class="math inline">\(c\)</span>, there are many different ways to satisfy these inequalities! To illustrate this, we’ll look at two different approaches here.</p>
|
||||
<p><strong>Approach 1</strong>: focus on choosing <span class="math inline">\(n_0\)</span>.</p>
|
||||
<p>It turns out we can satisfy the three inequalities even if <span class="math inline">\(c_1 = c_2 = c_3 = 1\)</span>:</p>
|
||||
<ul>
|
||||
<li><span class="math inline">\(n^3 \leq n^3\)</span> is always true (so for all <span class="math inline">\(n \geq 0\)</span>).</li>
|
||||
<li><span class="math inline">\(100n \leq n^3\)</span> when <span class="math inline">\(n \geq 10\)</span>.</li>
|
||||
<li><span class="math inline">\(5000 \leq n^3\)</span> when <span class="math inline">\(n \geq \sqrt[3]{5000} \approx 17.1\)</span></li>
|
||||
</ul>
|
||||
<p>We can pick <span class="math inline">\(n_0\)</span> to be the largest of the lower bounds on <span class="math inline">\(n\)</span>, <span class="math inline">\(\sqrt[3]{5000}\)</span>, and then these three inequalities will be satisfied!</p>
|
||||
<p><strong>Approach 2</strong>: focus on choosing <span class="math inline">\(c\)</span>.</p>
|
||||
<p>Another approach is to pick <span class="math inline">\(c_1\)</span>, <span class="math inline">\(c_2\)</span>, and <span class="math inline">\(c_3\)</span> to make the right-hand sides large enough to satisfy the inequalities.</p>
|
||||
<ul>
|
||||
<li><span class="math inline">\(n^3 \leq c_1 n^3\)</span> when <span class="math inline">\(c_1 = 1\)</span>.</li>
|
||||
<li><span class="math inline">\(100n \leq c_2 n^3\)</span> when <span class="math inline">\(c_2 = 100\)</span>.</li>
|
||||
<li><span class="math inline">\(5000 \leq c_3 n^3\)</span> when <span class="math inline">\(c_3 = 5000\)</span>, <em>as long as <span class="math inline">\(n \geq 1\)</span></em>.</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="proof">
|
||||
<p>(<em>Using Approach 1</em>) Let <span class="math inline">\(c = 3\)</span> and <span class="math inline">\(n_0 = \sqrt[3]{5000}\)</span>. Let <span class="math inline">\(n \in \N\)</span>, and assume that <span class="math inline">\(n \geq n_0\)</span>. We want to show that <span class="math inline">\(n^3 + 100n + 5000 \leq c n^3\)</span>.</p>
|
||||
<p>First, we prove three simpler inequalities:</p>
|
||||
<ul>
|
||||
<li><span class="math inline">\(n^3 \leq n^3\)</span> (since the two quantities are equal).</li>
|
||||
<li>Since <span class="math inline">\(n \geq n_0 \geq 10\)</span>, we know that <span class="math inline">\(n^2 \geq 100\)</span>, and so <span class="math inline">\(n^3 \geq 100n\)</span>.</li>
|
||||
<li>Since <span class="math inline">\(n \geq n_0\)</span>, we know that <span class="math inline">\(n^3 \geq n_0^3 = 5000\)</span>.</li>
|
||||
</ul>
|
||||
<p>Adding these three inequalities gives us: <span class="math display">\[n^3 + 100n + 5000 \leq n^3 + n^3 + n^3 = c n^3.\]</span></p>
|
||||
</div>
|
||||
<div class="proof">
|
||||
<p>(<em>Using Approach 2</em>) Let <span class="math inline">\(c = 5101\)</span> and <span class="math inline">\(n_0 = 1\)</span>. Let <span class="math inline">\(n \in \N\)</span>, and assume that <span class="math inline">\(n \geq n_0\)</span>. We want to show that <span class="math inline">\(n^3 + 100n + 5000 \leq c n^3\)</span>.</p>
|
||||
<p>First, we prove three simpler inequalities:</p>
|
||||
<ul>
|
||||
<li><span class="math inline">\(n^3 \leq n^3\)</span> (since the two quantities are equal).</li>
|
||||
<li>Since <span class="math inline">\(n \in \N\)</span>, we know that <span class="math inline">\(n \leq n^3\)</span>, and so <span class="math inline">\(100n \leq 100n^3\)</span>.</li>
|
||||
<li>Since <span class="math inline">\(1 \leq n\)</span>, we know that <span class="math inline">\(1 \leq n^3\)</span>, and then multiplying both sides by 5000 gives us <span class="math inline">\(5000 \leq 5000n^3\)</span>.</li>
|
||||
</ul>
|
||||
<p>Adding these three inequalities gives us: <span class="math display">\[n^3 + 100n + 5000 \leq n^3 + 100n^3 + 5000n^3 = 5101 n^3 = c n^3.\]</span></p>
|
||||
</div>
|
||||
</div>
|
||||
<!--div exercise>
|
||||
<div questions data-series="chapter5">
|
||||
\item
|
||||
Let $f: \N \to \R^{\ge 0}$, and let $y \in \R^+$ be an arbitrary positive real
|
||||
number.
|
||||
Prove that if $f \in \cO(y)$, then $f \in \cO(1)$ (this is why we write
|
||||
$\cO(1)$ and usually never see $\cO(2)$ or $\cO(110)$).
|
||||
</div>
|
||||
</div-->
|
||||
<footer>
|
||||
<a href="https://www.teach.cs.toronto.edu/~csc110y/fall/notes/">CSC110 Course Notes Home</a>
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,219 @@
|
||||
<!DOCTYPE html>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="generator" content="pandoc" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||||
<title>8.3 Big-O, Omega, and Theta</title>
|
||||
<style>
|
||||
code{white-space: pre-wrap;}
|
||||
span.smallcaps{font-variant: small-caps;}
|
||||
span.underline{text-decoration: underline;}
|
||||
div.column{display: inline-block; vertical-align: top; width: 50%;}
|
||||
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||||
ul.task-list{list-style: none;}
|
||||
</style>
|
||||
<link rel="stylesheet" href="../tufte.css" />
|
||||
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js" type="text/javascript"></script>
|
||||
<!--[if lt IE 9]>
|
||||
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div style="display:none">
|
||||
\(
|
||||
\newcommand{\NOT}{\neg}
|
||||
\newcommand{\AND}{\wedge}
|
||||
\newcommand{\OR}{\vee}
|
||||
\newcommand{\XOR}{\oplus}
|
||||
\newcommand{\IMP}{\Rightarrow}
|
||||
\newcommand{\IFF}{\Leftrightarrow}
|
||||
\newcommand{\TRUE}{\text{True}\xspace}
|
||||
\newcommand{\FALSE}{\text{False}\xspace}
|
||||
\newcommand{\IN}{\,{\in}\,}
|
||||
\newcommand{\NOTIN}{\,{\notin}\,}
|
||||
\newcommand{\TO}{\rightarrow}
|
||||
\newcommand{\DIV}{\mid}
|
||||
\newcommand{\NDIV}{\nmid}
|
||||
\newcommand{\MOD}[1]{\pmod{#1}}
|
||||
\newcommand{\MODS}[1]{\ (\text{mod}\ #1)}
|
||||
\newcommand{\N}{\mathbb N}
|
||||
\newcommand{\Z}{\mathbb Z}
|
||||
\newcommand{\Q}{\mathbb Q}
|
||||
\newcommand{\R}{\mathbb R}
|
||||
\newcommand{\C}{\mathbb C}
|
||||
\newcommand{\cA}{\mathcal A}
|
||||
\newcommand{\cB}{\mathcal B}
|
||||
\newcommand{\cC}{\mathcal C}
|
||||
\newcommand{\cD}{\mathcal D}
|
||||
\newcommand{\cE}{\mathcal E}
|
||||
\newcommand{\cF}{\mathcal F}
|
||||
\newcommand{\cG}{\mathcal G}
|
||||
\newcommand{\cH}{\mathcal H}
|
||||
\newcommand{\cI}{\mathcal I}
|
||||
\newcommand{\cJ}{\mathcal J}
|
||||
\newcommand{\cL}{\mathcal L}
|
||||
\newcommand{\cK}{\mathcal K}
|
||||
\newcommand{\cN}{\mathcal N}
|
||||
\newcommand{\cO}{\mathcal O}
|
||||
\newcommand{\cP}{\mathcal P}
|
||||
\newcommand{\cQ}{\mathcal Q}
|
||||
\newcommand{\cS}{\mathcal S}
|
||||
\newcommand{\cT}{\mathcal T}
|
||||
\newcommand{\cV}{\mathcal V}
|
||||
\newcommand{\cW}{\mathcal W}
|
||||
\newcommand{\cZ}{\mathcal Z}
|
||||
\newcommand{\emp}{\emptyset}
|
||||
\newcommand{\bs}{\backslash}
|
||||
\newcommand{\floor}[1]{\left \lfloor #1 \right \rfloor}
|
||||
\newcommand{\ceil}[1]{\left \lceil #1 \right \rceil}
|
||||
\newcommand{\abs}[1]{\left | #1 \right |}
|
||||
\newcommand{\xspace}{}
|
||||
\newcommand{\proofheader}[1]{\underline{\textbf{#1}}}
|
||||
\)
|
||||
</div>
|
||||
<header id="title-block-header">
|
||||
<h1 class="title">8.3 Big-O, Omega, and Theta</h1>
|
||||
</header>
|
||||
<section>
|
||||
<p>Big-O is a useful way of describing the long-term growth behaviour of functions, but its definition is limited in that it is not required to be an exact description of growth. After all, the key inequality <span class="math inline">\(g(n) \leq c f(n)\)</span> can be satisfied even if <span class="math inline">\(f\)</span> grows much, <em>much</em> faster than <span class="math inline">\(g\)</span>. For example, we could say that <span class="math inline">\(n + 10 \in \cO(n^{100})\)</span> according to our definition, but this is not necessarily informative.</p>
|
||||
<p>In other words, the definition of Big-O allows us to express <em>upper bounds</em> on the growth of a function, but does not allow us to distinguish between an upper bound that is tight and one that vastly overestimates the rate of growth.</p>
|
||||
<p>In this section, we will introduce the final new pieces of notation for this chapter, which allow us to express tight bounds on the growth of a function.</p>
|
||||
<h2 id="omega-and-theta">Omega and Theta</h2>
|
||||
<div class="definition">
|
||||
<p>Let <span class="math inline">\(f, g : \N \TO \R^{\ge 0}\)</span>. We say that <strong><span class="math inline">\(g\)</span> is Omega of <span class="math inline">\(f\)</span></strong> when there exist constants <span class="math inline">\(c, n_0 \in \R^+\)</span> such that for all <span class="math inline">\(n \in \N\)</span>, if <span class="math inline">\(n \geq n_0\)</span>, then <span class="math inline">\(g(n) \geq c \cdot f(n)\)</span>. In this case, we can also write <span class="math inline">\(g \in \Omega(f)\)</span>.</p>
|
||||
</div>
|
||||
<p>You can think of Omega as the dual of Big-O: when <span class="math inline">\(g \in \Omega(f)\)</span>, then <span class="math inline">\(f\)</span> is a <em>lower</em> bound on the growth rate of <span class="math inline">\(g\)</span>. For example, we can use the definition to prove that <span class="math inline">\(n^2 - n \in \Omega(n)\)</span>.</p>
|
||||
<p>We can now express a bound that is tight for a function’s growth rate quite elegantly by combining Big-O and Omega: if <span class="math inline">\(f\)</span> is asymptotically both a lower and upper bound for <span class="math inline">\(g\)</span>, then <span class="math inline">\(g\)</span> must grow at the same rate as <span class="math inline">\(f\)</span>.</p>
|
||||
<div class="definition" data-terms="Omega, Theta">
|
||||
<p>Let <span class="math inline">\(f, g : \N \TO \R^{\ge 0}\)</span>. We say that <strong><span class="math inline">\(g\)</span> is Theta of <span class="math inline">\(f\)</span></strong> when <span class="math inline">\(g\)</span> is both Big-O of <span class="math inline">\(f\)</span> and Omega of <span class="math inline">\(f\)</span>. In this case, we can write <span class="math inline">\(g \in \Theta(f)\)</span>, and say that <span class="math inline">\(f\)</span> is a <strong>tight bound</strong> on <span class="math inline">\(g\)</span>.<label for="sn-0" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-0" class="margin-toggle"/><span class="sidenote">Most of the time, when people say “Big-O” they actually mean Theta, i.e., a Big-O upper bound is meant to be the tight one, because we rarely say upper bounds that overestimate the rate of growth. However, in this course we will always use <span class="math inline">\(\Theta\)</span> when we mean tight bounds, because we will see some cases where coming up with tight bounds isn’t easy.</span></p>
|
||||
<p>Equivalently, <span class="math inline">\(g\)</span> is Theta of <span class="math inline">\(f\)</span> when there exist constants <span class="math inline">\(c_1, c_2, n_0 \in \R^+\)</span> such that for all <span class="math inline">\(n \in \N\)</span>, if <span class="math inline">\(n \geq n_0\)</span> then <span class="math inline">\(c_1 f(n) \leq g(n) \leq c_2 f(n)\)</span>.</p>
|
||||
</div>
|
||||
<p>When we are comparing function growth rates, we typically look for a “Theta bound”, as this means that the two functions have the same approximate rate of growth, not just that one is larger than the other. For example, it is possible to prove that <span class="math inline">\(10n + 5 \in \Theta(n)\)</span>, but <span class="math inline">\(10n + 5 \notin \Theta(n^2)\)</span>.<label for="sn-1" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-1" class="margin-toggle"/><span class="sidenote"> Both of these are good exercises to prove, using the above definitions!</span></p>
|
||||
<!-- <div class="example">
|
||||
Let $f(n) = n^2$ and $g(n) = n + 10$. Then $g \in \cO(f)$, but
|
||||
$g \notin \Theta(f)$. That is, $f$ is an upper bound for the growth
|
||||
rate of $g$, but it is not a tight upper bound.
|
||||
</div>
|
||||
|
||||
<div exercise>
|
||||
<div questions data-series="chapter5">
|
||||
\item
|
||||
Prove the statement in the previous example.
|
||||
Note that the correct translation uses an **AND**, so you'll actually need to prove two different statements here.
|
||||
</div>
|
||||
</div> -->
|
||||
<h2 id="a-special-case-co1-omega1-and-theta1">A special case: <span class="math inline">\(\cO(1)\)</span>, <span class="math inline">\(\Omega(1)\)</span>, and <span class="math inline">\(\Theta(1)\)</span></h2>
|
||||
<p>So far, we have seen Big-O expressions like <span class="math inline">\(\cO(n)\)</span> and <span class="math inline">\(\cO(n^2)\)</span>, where the function in parentheses has grown to infinity. However, not every function takes on larger and larger values as its input grows. Some functions are <em>bounded</em>, meaning they never take on a value larger than some fixed constant.</p>
|
||||
<p>For example, consider the constant function <span class="math inline">\(f(n) = 1\)</span>, which always outputs the value <span class="math inline">\(1\)</span>, regardless of the value of <span class="math inline">\(n\)</span>. What would it mean to say that a function <span class="math inline">\(g\)</span> is Big-O of this <span class="math inline">\(f\)</span>? Let’s unpack the definition of Big-O to find out.</p>
|
||||
<p><span class="math display">\[\begin{align*}
|
||||
& g \in \cO(f) \\
|
||||
& \exists c, n_0 \in \R^+,~ \forall n \in \N,~ n \geq n_0 \IMP g(n) \leq c \cdot f(n) \\
|
||||
& \exists c, n_0 \in \R^+,~ \forall n \in \N,~ n \geq n_0 \IMP g(n) \leq c \tag{since $f(n) = 1$}
|
||||
\end{align*}\]</span></p>
|
||||
<p>In other words, there exists a constant <span class="math inline">\(c\)</span> such that <span class="math inline">\(g(n)\)</span> is eventually always less than or equal to <span class="math inline">\(c\)</span>. We say that such functions <span class="math inline">\(g\)</span> are <strong>asymptotically bounded</strong> with respect to their input, and write <span class="math inline">\(g \in \cO(1)\)</span> to represent this.</p>
|
||||
<p>Similarly, we use <span class="math inline">\(g \in \Omega(1)\)</span> to express that functions are greater than or equal to some constant <span class="math inline">\(c\)</span>. You might wonder why we would ever say this—don’t all functions satisfy this property? While the functions we’ll be studying in later chapters in this section are generally going to be <span class="math inline">\(\geq 1\)</span>, this is not true for all mathematical functions. For example, the function <span class="math inline">\(g(n) = \frac{1}{n + 1}\)</span> is <span class="math inline">\(\cO(1)\)</span>, but <em>not</em> <span class="math inline">\(\Omega(1)\)</span>.<label for="sn-2" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-2" class="margin-toggle"/><span class="sidenote"> More generally, any function <span class="math inline">\(g\)</span> where <span class="math inline">\(\lim_{n \to \infty} g(n) = 0\)</span> is not <span class="math inline">\(\Omega(1)\)</span>.</span></p>
|
||||
<p>On the other hand, the function <span class="math inline">\(g(n) = n^2\)</span> is <span class="math inline">\(\Omega(1)\)</span> but not <span class="math inline">\(\cO(1)\)</span>. So we reserve <span class="math inline">\(\Theta(1)\)</span> to refer to the functions that are both <span class="math inline">\(\cO(1)\)</span> and <span class="math inline">\(\Omega(1)\)</span>.</p>
|
||||
<h2 id="section:asymptotic-properties">Properties of Big-O, Omega, and Theta</h2>
|
||||
<p>If we had you always write chains of inequalities to prove that one function is Big-O/Omega/Theta of another, that would get quite tedious rather quickly. Instead, in this section we will prove some properties of this definition which are extremely useful for combining functions together under this definition. These properties can save you quite a lot of work in the long run. We’ll illustrate the proof of one of these properties here; most of the others can be proved in a similar manner, while a few are most easily proved using some techniques from calculus.<label for="sn-3" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-3" class="margin-toggle"/><span class="sidenote">We discuss the connection between calculus and asymptotic notation in the following section, but this is <em>not</em> a required part of CSC110.</span></p>
|
||||
<h3 id="elementary-functions">Elementary functions</h3>
|
||||
<p>The following theorem tells us how to compare four different types of “elementary” functions: constant functions, logarithms, powers of <span class="math inline">\(n\)</span>, and exponential functions.</p>
|
||||
<div id="elementary-function-hierarchy" class="theorem" data-label="Elementary function hierarchy">
|
||||
<p>(<em>Elementary function growth hierarchy</em>)</p>
|
||||
<p>For all <span class="math inline">\(a, b \in \R^+\)</span>, the following statements are true:</p>
|
||||
<ol type="1">
|
||||
<li>If <span class="math inline">\(a > 1\)</span> and <span class="math inline">\(b > 1\)</span>, then <span class="math inline">\(\log_a n \in \Theta(\log_b n)\)</span>.</li>
|
||||
<li>If <span class="math inline">\(a < b\)</span>, then <span class="math inline">\(n^a \in \cO(n^b)\)</span> and <span class="math inline">\(n^a \notin \Omega(n^b)\)</span>.</li>
|
||||
<li>If <span class="math inline">\(a < b\)</span>, then <span class="math inline">\(a^n \in \cO(b^n)\)</span> and <span class="math inline">\(a^n \notin \Omega(b^n)\)</span>.</li>
|
||||
<li>If <span class="math inline">\(a > 1\)</span>, then <span class="math inline">\(1 \in \cO(\log_a n)\)</span> and <span class="math inline">\(1 \notin \Omega(\log_a n)\)</span>.</li>
|
||||
<li><span class="math inline">\(\log_a n \in \cO(n^b)\)</span> and <span class="math inline">\(\log_a n \notin \Omega(n^b)\)</span>.</li>
|
||||
<li>If <span class="math inline">\(b > 1\)</span>, then <span class="math inline">\(n^a \in \cO(b^n)\)</span> and <span class="math inline">\(n^a \notin \Omega(b^n)\)</span>.</li>
|
||||
</ol>
|
||||
</div>
|
||||
<p>And here is a handy figure to show the progression of functions toward longer running times:</p>
|
||||
<p><img src="images/elementary_function_hierarchy.png" /></p>
|
||||
<h3 id="basic-properties">Basic properties</h3>
|
||||
<div class="theorem" data-label="Reflexivity">
|
||||
<p>For all <span class="math inline">\(f : \N \to \R^{\geq 0}\)</span>, <span class="math inline">\(f \in \Theta(f)\)</span>.</p>
|
||||
</div>
|
||||
<div class="theorem" data-label="Quasi-symmetry">
|
||||
<p>For all <span class="math inline">\(f, g : \N \to \R^{\geq 0}\)</span>, <span class="math inline">\(g \in \cO(f)\)</span> if and only if <span class="math inline">\(f \in \Omega(g)\)</span>.<label for="sn-4" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-4" class="margin-toggle"/><span class="sidenote">As a consequence of this, <span class="math inline">\(g \in \Theta(f)\)</span> if and only if <span class="math inline">\(f \in \Theta(g)\)</span>.</span></p>
|
||||
</div>
|
||||
<div class="theorem" data-label="Transitivity">
|
||||
<p>For all <span class="math inline">\(f, g, h : \N \to \R^{\geq 0}\)</span>:</p>
|
||||
<ul>
|
||||
<li>If <span class="math inline">\(f \in \cO(g)\)</span> and <span class="math inline">\(g \in \cO(h)\)</span>, then <span class="math inline">\(f \in \cO(h)\)</span>.</li>
|
||||
<li>If <span class="math inline">\(f \in \Omega(g)\)</span> and <span class="math inline">\(g \in \Omega(h)\)</span>, then <span class="math inline">\(f \in \Omega(h)\)</span>.</li>
|
||||
<li>If <span class="math inline">\(f \in \Theta(g)\)</span> and <span class="math inline">\(g \in \Theta(h)\)</span>, then <span class="math inline">\(f \in \Theta(h)\)</span>.<label for="sn-5" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-5" class="margin-toggle"/><span class="sidenote"> Exercise: prove this using the first two.</span></li>
|
||||
</ul>
|
||||
</div>
|
||||
<h3 id="operations-on-functions">Operations on functions</h3>
|
||||
<div class="definition" data-terms="sum of functions">
|
||||
<p>Let <span class="math inline">\(f, g : \N \TO \R^{\ge 0}\)</span>. We can define the <strong>sum of <span class="math inline">\(f\)</span> and <span class="math inline">\(g\)</span></strong> as the function <span class="math inline">\(f + g : \N \TO \R^{\ge 0}\)</span> such that <span class="math display">\[\forall n \in \N,~ (f + g)(n) = f(n) + g(n).\]</span></p>
|
||||
</div>
|
||||
<div class="theorem" data-label="Sum of functions">
|
||||
<p> For all <span class="math inline">\(f, g, h : \N \to \R^{\geq 0}\)</span>, the following hold:</p>
|
||||
<ol type="1">
|
||||
<li>If <span class="math inline">\(f \in \cO(h)\)</span> and <span class="math inline">\(g \in \cO(h)\)</span>, then <span class="math inline">\(f + g \in \cO(h)\)</span>.</li>
|
||||
<li>If <span class="math inline">\(f \in \Omega(h)\)</span>, then <span class="math inline">\(f + g \in \Omega(h)\)</span>.</li>
|
||||
<li>If <span class="math inline">\(f \in \Theta(h)\)</span> and <span class="math inline">\(g \in \cO(h)\)</span>, then <span class="math inline">\(f + g \in \Theta(h)\)</span>.<label for="sn-6" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-6" class="margin-toggle"/><span class="sidenote"> Exercise: prove this using the first two.</span></li>
|
||||
</ol>
|
||||
<p>We’ll prove the first of these statements.</p>
|
||||
<div class="translation">
|
||||
<p><span class="math display">\[\forall f, g, h : \N \TO \R^{\ge 0},~ \big(f \in \cO(h) \AND g \in \cO(h)\big) \IMP f + g \in \cO(h).\]</span></p>
|
||||
</div>
|
||||
<div class="discussion">
|
||||
<p>This is similar in spirit to the divisibility proofs we did in <a href="../06-proofs/02-number-theory-proofs.html">Section 6.2</a>, which used a term (divisibility) that contained a quantifier.<label for="sn-7" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-7" class="margin-toggle"/><span class="sidenote">The definition of Big-O here has <em>three</em> quantifiers, but the idea is the same.</span> Here, we need to assume that <span class="math inline">\(f\)</span> and <span class="math inline">\(g\)</span> are both Big-O of <span class="math inline">\(h\)</span>, and prove that <span class="math inline">\(f + g\)</span> is also Big-O of <span class="math inline">\(h\)</span>.</p>
|
||||
<p>Assuming <span class="math inline">\(f \in \cO(h)\)</span> tells us there exist positive real numbers <span class="math inline">\(c_1\)</span> and <span class="math inline">\(n_1\)</span> such that for all <span class="math inline">\(n \in \N\)</span>, if <span class="math inline">\(n \geq n_1\)</span> then <span class="math inline">\(f(n) \leq c_1 \cdot h(n)\)</span>. There similarly exist <span class="math inline">\(c_2\)</span> and <span class="math inline">\(n_2\)</span> such that <span class="math inline">\(g(n) \leq c_2 \cdot h(n)\)</span> whenever <span class="math inline">\(n \geq n_2\)</span>. <em>Warning:</em> we can’t assume that <span class="math inline">\(c_1 = c_2\)</span> or <span class="math inline">\(n_1 = n_2\)</span>, or any other relationship between these two sets of variables.</p>
|
||||
<p>We want to prove that there exist <span class="math inline">\(c, n_0 \in \R^+\)</span> such that for all <span class="math inline">\(n \in \N\)</span>, if <span class="math inline">\(n \geq n_0\)</span> then <span class="math inline">\(f(n) + g(n) \leq c \cdot h(n)\)</span>.</p>
|
||||
<p>The forms of the inequalities we can assume—<span class="math inline">\(f(n) \leq c_1 h(n)\)</span>, <span class="math inline">\(g(n) \leq c_2 h(n)\)</span>—and the final inequality are identical, and in particular the left-hand side suggests that we just need to add the two given inequalities together to get the third. We just need to make sure that both given inequalities hold by choosing <span class="math inline">\(n_0\)</span> to be large enough, and let <span class="math inline">\(c\)</span> be large enough to take into account both <span class="math inline">\(c_1\)</span> and <span class="math inline">\(c_2\)</span>.</p>
|
||||
</div>
|
||||
<div class="proof">
|
||||
<p>Let <span class="math inline">\(f, g, h : \N \TO \R^{\ge 0}\)</span>, and assume <span class="math inline">\(f \in \cO(h)\)</span> and <span class="math inline">\(g \in \cO(h)\)</span>. By these assumptions, there exist <span class="math inline">\(c_1, c_2, n_1, n_2 \in \R^+\)</span> such that for all <span class="math inline">\(n \in \N\)</span>,</p>
|
||||
<ul>
|
||||
<li>if <span class="math inline">\(n \geq n_1\)</span>, then <span class="math inline">\(f(n) \leq c_1 \cdot h(n)\)</span>, and</li>
|
||||
<li>if <span class="math inline">\(n \geq n_2\)</span>, then <span class="math inline">\(g(n) \leq c_2 \cdot h(n)\)</span>.</li>
|
||||
</ul>
|
||||
<p>We want to prove that <span class="math inline">\(f + g \in \cO(h)\)</span>, i.e., that there exist <span class="math inline">\(c, n_0 \in \R^+\)</span> such that for all <span class="math inline">\(n \in \N\)</span>, if <span class="math inline">\(n \geq n_0\)</span> then <span class="math inline">\(f(n) + g(n) \leq c \cdot h(n)\)</span>.</p>
|
||||
<p>Let <span class="math inline">\(n_0 = \max \{n_1, n_2\}\)</span> and <span class="math inline">\(c = c_1 + c_2\)</span>. Let <span class="math inline">\(n \in \N\)</span>, and assume that <span class="math inline">\(n \geq n_0\)</span>. We now want to prove that <span class="math inline">\(f(n) + g(n) \leq c \cdot h(n)\)</span>.</p>
|
||||
<p>Since <span class="math inline">\(n_0 \geq n_1\)</span> and <span class="math inline">\(n_0 \geq n_2\)</span>, we know that <span class="math inline">\(n\)</span> is greater than or equal to <span class="math inline">\(n_1\)</span> and <span class="math inline">\(n_2\)</span> as well. Then using the Big-O assumptions, <span class="math display">\[\begin{align*}
|
||||
f(n) &\leq c_1 \cdot h(n) \\
|
||||
g(n) &\leq c_2 \cdot h(n)
|
||||
\end{align*}\]</span></p>
|
||||
<p>Adding these two inequalities together yields</p>
|
||||
<p><span class="math display">\[f(n) + g(n) \leq c_1 h(n) + c_2 h(n) = (c_1 + c_2) h(n) = c \cdot h(n).\]</span></p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="theorem" data-label="Multiplication by a constant">
|
||||
<p>For all <span class="math inline">\(f : \N \to \R^{\geq 0}\)</span> and all <span class="math inline">\(a \in \R^+\)</span>, <span class="math inline">\(a \cdot f \in \Theta(f)\)</span>.</p>
|
||||
</div>
|
||||
<div class="theorem" data-label="Product of functions">
|
||||
<p>For all <span class="math inline">\(f_1, f_2, g_1, g_2 : \N \to \R^{\geq 0}\)</span>, if <span class="math inline">\(g_1 \in \cO(f_1)\)</span> and <span class="math inline">\(g_2 \in \cO(f_2)\)</span>, then <span class="math inline">\(g_1 \cdot g_2 \in \cO(f_1 \cdot f_2)\)</span>. <em>Moreover</em>, the statement is still true if you replace Big-O with Omega, or if you replace Big-O with Theta.</p>
|
||||
</div>
|
||||
<div class="theorem" data-label="Floor and ceiling">
|
||||
<p>For all <span class="math inline">\(f : \N \to \R^{\geq 0}\)</span>, if <span class="math inline">\(f(n)\)</span> is eventually greater than or equal to <span class="math inline">\(1\)</span>, then <span class="math inline">\(\floor{f} \in \Theta(f)\)</span> and <span class="math inline">\(\ceil{f} \in \Theta(f)\)</span>.</p>
|
||||
</div>
|
||||
<h3 id="properties-from-calculus">Properties from calculus</h3>
|
||||
<p>[<em>Note: this subsection is <strong>not</strong> part of the require course material for CSC110. It is presented mainly for the nice connection between Big-O notation and calculus.</em>]</p>
|
||||
<p>Our asymptotic notation of <span class="math inline">\(\cO\)</span>, <span class="math inline">\(\Omega\)</span>, and <span class="math inline">\(\Theta\)</span> are concerned with the comparing the <em>long-term behaviour</em> of two functions. It turns out that the concept of “long-term behaviour” is captured in another object of mathematical study, familiar to us from calculus: the <em>limit</em> of the function as its input approaches infinity.</p>
|
||||
<p>Let <span class="math inline">\(f: \N \to \R\)</span> and <span class="math inline">\(L \in \R\)</span>. We have the following two definitions:<label for="sn-8" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-8" class="margin-toggle"/><span class="sidenote">We’re restricting our attention here to functions with domain <span class="math inline">\(\N\)</span> because that’s our focus in computer science.</span> <span class="math display">\[
|
||||
\lim_{n \to \infty} f(n) = L:~ \forall \epsilon \in \R^+,~ \exists n_0 \in \N,~ \forall n \in \N,~ n \geq n_0 \IMP |f(n) - L| < \epsilon
|
||||
\]</span> <span class="math display">\[
|
||||
\lim_{n \to \infty} f(n) = \infty:~ \forall M \in \R^+,~ \exists n_0 \in \N,~ \forall n \in \N,~ n \geq n_0 \IMP f(n) > M
|
||||
\]</span></p>
|
||||
<p>Using just these definitions and the definitions of our asymptotic symbols <span class="math inline">\(\cO\)</span>, <span class="math inline">\(\Omega\)</span>, and <span class="math inline">\(\Theta\)</span>, we can prove the following pretty remarkable results:</p>
|
||||
<div class="theorem" data-label="Limits and Big-O">
|
||||
<p>For all <span class="math inline">\(f, g: \N \to \R^{\geq 0}\)</span>, if <span class="math inline">\(g(n) \neq 0\)</span> for all <span class="math inline">\(n \in \N\)</span>, then the following statements hold:</p>
|
||||
<ol type="i">
|
||||
<li>If there exists <span class="math inline">\(L \in \R^+\)</span> such that <span class="math inline">\(\lim_{n \to \infty} f(n)/g(n) = L\)</span>, then <span class="math inline">\(g \in \Omega(f)\)</span> and <span class="math inline">\(g \in \cO(f)\)</span>. (In other words, <span class="math inline">\(g \in \Theta(f)\)</span>.)</li>
|
||||
<li>If <span class="math inline">\(\lim_{n \to \infty} f(n)/g(n) = 0\)</span>, then <span class="math inline">\(f \in \cO(g)\)</span> and <span class="math inline">\(g \notin \cO(f)\)</span>.</li>
|
||||
<li>If <span class="math inline">\(\lim_{n \to \infty} f(n)/g(n) = \infty\)</span>, then <span class="math inline">\(g \in \cO(f)\)</span> and <span class="math inline">\(f \notin \cO(g)\)</span>.</li>
|
||||
</ol>
|
||||
</div>
|
||||
<p>Proving this theorem is actually a very good (lengthy) exercise for a CSC110 student; they involve keeping track of variables and manipulating inequalities, two key skills you’re developing in this course! And they do tend to be useful in practice (although again, not for this course) to proving asymptotic bounds like <span class="math inline">\(n^2 \in \cO(1.01^n)\)</span>. But note that the converse of these statements is not true; for example, it is possible (and another nice exercise) to find functions <span class="math inline">\(f\)</span> and <span class="math inline">\(g\)</span> such that <span class="math inline">\(g \in \Theta(f)\)</span>, but <span class="math inline">\(\lim_{n \to \infty} f(n)/g(n)\)</span> is undefined.</p>
|
||||
</section>
|
||||
<footer>
|
||||
<a href="https://www.teach.cs.toronto.edu/~csc110y/fall/notes/">CSC110 Course Notes Home</a>
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,287 @@
|
||||
<!DOCTYPE html>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="generator" content="pandoc" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||||
<title>8.4 Analyzing Algorithm Running Time</title>
|
||||
<style>
|
||||
code{white-space: pre-wrap;}
|
||||
span.smallcaps{font-variant: small-caps;}
|
||||
span.underline{text-decoration: underline;}
|
||||
div.column{display: inline-block; vertical-align: top; width: 50%;}
|
||||
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||||
ul.task-list{list-style: none;}
|
||||
pre > code.sourceCode { white-space: pre; position: relative; }
|
||||
pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
|
||||
pre > code.sourceCode > span:empty { height: 1.2em; }
|
||||
code.sourceCode > span { color: inherit; text-decoration: inherit; }
|
||||
div.sourceCode { margin: 1em 0; }
|
||||
pre.sourceCode { margin: 0; }
|
||||
@media screen {
|
||||
div.sourceCode { overflow: auto; }
|
||||
}
|
||||
@media print {
|
||||
pre > code.sourceCode { white-space: pre-wrap; }
|
||||
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
|
||||
}
|
||||
pre.numberSource code
|
||||
{ counter-reset: source-line 0; }
|
||||
pre.numberSource code > span
|
||||
{ position: relative; left: -4em; counter-increment: source-line; }
|
||||
pre.numberSource code > span > a:first-child::before
|
||||
{ content: counter(source-line);
|
||||
position: relative; left: -1em; text-align: right; vertical-align: baseline;
|
||||
border: none; display: inline-block;
|
||||
-webkit-touch-callout: none; -webkit-user-select: none;
|
||||
-khtml-user-select: none; -moz-user-select: none;
|
||||
-ms-user-select: none; user-select: none;
|
||||
padding: 0 4px; width: 4em;
|
||||
color: #aaaaaa;
|
||||
}
|
||||
pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; }
|
||||
div.sourceCode
|
||||
{ }
|
||||
@media screen {
|
||||
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
|
||||
}
|
||||
code span.al { color: #ff0000; font-weight: bold; } /* Alert */
|
||||
code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
|
||||
code span.at { color: #7d9029; } /* Attribute */
|
||||
code span.bn { color: #40a070; } /* BaseN */
|
||||
code span.bu { } /* BuiltIn */
|
||||
code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
|
||||
code span.ch { color: #4070a0; } /* Char */
|
||||
code span.cn { color: #880000; } /* Constant */
|
||||
code span.co { color: #60a0b0; font-style: italic; } /* Comment */
|
||||
code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
|
||||
code span.do { color: #ba2121; font-style: italic; } /* Documentation */
|
||||
code span.dt { color: #902000; } /* DataType */
|
||||
code span.dv { color: #40a070; } /* DecVal */
|
||||
code span.er { color: #ff0000; font-weight: bold; } /* Error */
|
||||
code span.ex { } /* Extension */
|
||||
code span.fl { color: #40a070; } /* Float */
|
||||
code span.fu { color: #06287e; } /* Function */
|
||||
code span.im { } /* Import */
|
||||
code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
|
||||
code span.kw { color: #007020; font-weight: bold; } /* Keyword */
|
||||
code span.op { color: #666666; } /* Operator */
|
||||
code span.ot { color: #007020; } /* Other */
|
||||
code span.pp { color: #bc7a00; } /* Preprocessor */
|
||||
code span.sc { color: #4070a0; } /* SpecialChar */
|
||||
code span.ss { color: #bb6688; } /* SpecialString */
|
||||
code span.st { color: #4070a0; } /* String */
|
||||
code span.va { color: #19177c; } /* Variable */
|
||||
code span.vs { color: #4070a0; } /* VerbatimString */
|
||||
code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
|
||||
</style>
|
||||
<link rel="stylesheet" href="../tufte.css" />
|
||||
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js" type="text/javascript"></script>
|
||||
<!--[if lt IE 9]>
|
||||
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div style="display:none">
|
||||
\(
|
||||
\newcommand{\NOT}{\neg}
|
||||
\newcommand{\AND}{\wedge}
|
||||
\newcommand{\OR}{\vee}
|
||||
\newcommand{\XOR}{\oplus}
|
||||
\newcommand{\IMP}{\Rightarrow}
|
||||
\newcommand{\IFF}{\Leftrightarrow}
|
||||
\newcommand{\TRUE}{\text{True}\xspace}
|
||||
\newcommand{\FALSE}{\text{False}\xspace}
|
||||
\newcommand{\IN}{\,{\in}\,}
|
||||
\newcommand{\NOTIN}{\,{\notin}\,}
|
||||
\newcommand{\TO}{\rightarrow}
|
||||
\newcommand{\DIV}{\mid}
|
||||
\newcommand{\NDIV}{\nmid}
|
||||
\newcommand{\MOD}[1]{\pmod{#1}}
|
||||
\newcommand{\MODS}[1]{\ (\text{mod}\ #1)}
|
||||
\newcommand{\N}{\mathbb N}
|
||||
\newcommand{\Z}{\mathbb Z}
|
||||
\newcommand{\Q}{\mathbb Q}
|
||||
\newcommand{\R}{\mathbb R}
|
||||
\newcommand{\C}{\mathbb C}
|
||||
\newcommand{\cA}{\mathcal A}
|
||||
\newcommand{\cB}{\mathcal B}
|
||||
\newcommand{\cC}{\mathcal C}
|
||||
\newcommand{\cD}{\mathcal D}
|
||||
\newcommand{\cE}{\mathcal E}
|
||||
\newcommand{\cF}{\mathcal F}
|
||||
\newcommand{\cG}{\mathcal G}
|
||||
\newcommand{\cH}{\mathcal H}
|
||||
\newcommand{\cI}{\mathcal I}
|
||||
\newcommand{\cJ}{\mathcal J}
|
||||
\newcommand{\cL}{\mathcal L}
|
||||
\newcommand{\cK}{\mathcal K}
|
||||
\newcommand{\cN}{\mathcal N}
|
||||
\newcommand{\cO}{\mathcal O}
|
||||
\newcommand{\cP}{\mathcal P}
|
||||
\newcommand{\cQ}{\mathcal Q}
|
||||
\newcommand{\cS}{\mathcal S}
|
||||
\newcommand{\cT}{\mathcal T}
|
||||
\newcommand{\cV}{\mathcal V}
|
||||
\newcommand{\cW}{\mathcal W}
|
||||
\newcommand{\cZ}{\mathcal Z}
|
||||
\newcommand{\emp}{\emptyset}
|
||||
\newcommand{\bs}{\backslash}
|
||||
\newcommand{\floor}[1]{\left \lfloor #1 \right \rfloor}
|
||||
\newcommand{\ceil}[1]{\left \lceil #1 \right \rceil}
|
||||
\newcommand{\abs}[1]{\left | #1 \right |}
|
||||
\newcommand{\xspace}{}
|
||||
\newcommand{\proofheader}[1]{\underline{\textbf{#1}}}
|
||||
\)
|
||||
</div>
|
||||
<header id="title-block-header">
|
||||
<h1 class="title">8.4 Analyzing Algorithm Running Time</h1>
|
||||
</header>
|
||||
<section>
|
||||
<p>Let us consider a very similar function to <code>print_integers</code> from the beginning of the chapter:</p>
|
||||
<div class="sourceCode" id="cb1"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1"></a><span class="kw">def</span> print_items(lst: <span class="bu">list</span>) <span class="op">-></span> <span class="va">None</span>:</span>
|
||||
<span id="cb1-2"><a href="#cb1-2"></a> <span class="cf">for</span> item <span class="kw">in</span> lst:</span>
|
||||
<span id="cb1-3"><a href="#cb1-3"></a> <span class="bu">print</span>(item)</span></code></pre></div>
|
||||
<p>Here, <code>print_items</code> takes a list as input instead, and so <span class="math inline">\(n\)</span> is equivalent to <code>len(lst)</code>.<label for="sn-0" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-0" class="margin-toggle"/><span class="sidenote"> For the remainder of this course, we will assume input size for a list is always its length, unless something else is specified.</span> How can we use our asymptotic notation to help us analyze the running time of this algorithm? Earlier, we said that the call to <code>print</code> took 1 “basic operation”, but is that true? The answer is, it doesn’t matter. By using asymptotic notation, we no longer need to worry about the constants involved, and so don’t need to worry about whether a single call to <code>print</code> counts as one or ten “basic operations”.</p>
|
||||
<p>Just as switching from measuring real time to counting “basic operations” allows us to ignore the computing environment in which the program runs, switching from an exact step count to asymptotic notation allows us to ignore machine- and programming language-dependent constants involved in the execution of the code. Having ignored all these external factors, our analysis will concentrate on how the <strong>size of the input</strong> influences the running time of a program, where we measure running time just using asymptotic notation, and not exact expressions.</p>
|
||||
<p><strong>Warning</strong>: the “size” of the input to a program can mean different things depending on the type of input, or even depending on the program itself. Whenever you perform a running time analysis, be sure to clearly state how you are measuring and representing input size.</p>
|
||||
<p>Because constants don’t matter, we will use a very coarse measure of “basic operation” to make our analysis as simple as possible. For our purposes, a basic operation (or step) is <strong>any block of code whose running time does not depend on the size of the input</strong>.<label for="sn-1" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-1" class="margin-toggle"/><span class="sidenote">To belabour the point a little, this depends on how we define input size. For integers, we usually will assume they have a fixed size in memory (e.g., 32 bits), which is why arithmetic operations take constant time. But of course if we allow numbers to grow infinitely, this is no longer true, and performing arithmetic operations will no longer take constant time.</span></p>
|
||||
<p>This includes all primitive language operations like most assignment statements, arithmetic calculations, and list and string indexing. The one major statement type which does not fit in this category is a function call—the running time of such statements depends on how long that particular function takes to run. We’ll revisit this in more detail later.</p>
|
||||
<h2 id="the-runtime-function">The runtime function</h2>
|
||||
<p>The running time of <code>print_items</code> depends <em>only</em> on the size of the input list, and not the contents of the list. That is, we expect that <code>print_items</code> takes the same amount of time on every list of length <span class="math inline">\(100\)</span>. We can make this a little more clear by introducing one piece of notation that will come in handy for the rest of the chapter.</p>
|
||||
<div class="definition" data-terms="input sets">
|
||||
<p>Let <code>func</code> be an algorithm. For every <span class="math inline">\(n \in \N\)</span>, we define the set <span class="math inline">\(\cI_{func, n}\)</span> to be the set of allowed inputs to <code>func</code> of size <span class="math inline">\(n\)</span>.</p>
|
||||
</div>
|
||||
<p>For example, <span class="math inline">\(\cI_{print\_items, 100}\)</span> is simply the set of all lists of length 100. <span class="math inline">\(\cI_{print\_items, 0}\)</span> is the set containing just one input: the empty list.</p>
|
||||
<p>We can restate our observation about <code>print_items</code> in terms of these sets: for all <span class="math inline">\(n \in \N\)</span>, every element of <span class="math inline">\(\cI_{print\_items, n}\)</span> has the <em>same</em> runtime when passed to <code>print_items</code>.</p>
|
||||
<div class="definition" data-terms="runtime function">
|
||||
<p>Let <code>func</code> be an algorithm whose runtime depends <em>only</em> on its input size. We define the <strong>running time function of <code>func</code></strong> as <span class="math inline">\(RT_{func}: \N \to \R^{\geq 0}\)</span>, where <span class="math inline">\(RT_{func}(n)\)</span> is equal to the running time of <code>func</code> when given an input of size <span class="math inline">\(n\)</span>.</p>
|
||||
<p>The goal of a <em>running time analysis</em> for <code>func</code> is to find a function <span class="math inline">\(f\)</span> (typically a simple elementary function) such that <span class="math inline">\(RT_{func} \in \Theta(f)\)</span>.</p>
|
||||
</div>
|
||||
<p>Our first technique for performing this runtime analysis follows four steps:</p>
|
||||
<ol type="1">
|
||||
<li>Identify the blocks of code which can be counted as a single basic operation, because they don’t depend on the input size.</li>
|
||||
<li>Identify any loops in the code, which cause basic operations to repeat. You’ll need to figure out how many times those loops run, based on the size of the input. Be <em>exact</em> when counting loop iterations.</li>
|
||||
<li>Use your observations from the previous two steps to come up with an expression for the number of basic operations used in this algorithm—i.e., find an exact expression for <span class="math inline">\(RT_{func}(n)\)</span>.</li>
|
||||
<li>Use the properties of asymptotic notation to find an elementary function <span class="math inline">\(f\)</span> such that <span class="math inline">\(RT_{func} \in \Theta(f(n))\)</span>.</li>
|
||||
</ol>
|
||||
<p>Because Theta expressions depend only on the fastest-growing term in a sum, <em>and</em> ignores constants, we don’t even need an exact, “correct” expression for the number of basic operations. This allows us to be rough with our analysis, but still get the correct Theta expression.</p>
|
||||
<div class="example">
|
||||
<p>Consider the function <code>print_items</code>. We define input size to be the <em>number of items of the input list</em>. Perform a runtime analysis of <code>print_items</code>.</p>
|
||||
<div class="sourceCode" id="cb2"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1"></a><span class="kw">def</span> print_items(lst: <span class="bu">list</span>) <span class="op">-></span> <span class="va">None</span>:</span>
|
||||
<span id="cb2-2"><a href="#cb2-2"></a> <span class="cf">for</span> item <span class="kw">in</span> lst:</span>
|
||||
<span id="cb2-3"><a href="#cb2-3"></a> <span class="bu">print</span>(item)</span></code></pre></div>
|
||||
<div class="analysis">
|
||||
<p>Let <span class="math inline">\(n\)</span> be the length of the input list <code>lst</code>.</p>
|
||||
<p>For this algorithm, each iteration of the loop can be counted as a single operation, because nothing in it (including the call to <code>print</code>) depends on the size of the input list.<label for="sn-2" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-2" class="margin-toggle"/><span class="sidenote">This is actually a little subtle. If we consider the size of individual list elements, it could be the case that some take a much longer time to print than others (imagine printing a string of one-thousand characters vs. the number <span class="math inline">\(5\)</span>). But by defining input size purely as the number of items, we are implicitly ignoring the size of the individual items. The running time of a call to <code>print</code> does <em>not</em> depend on the length of the input list.</span></p>
|
||||
<p>So the running time depends on the number of loop iterations. Since this is a for loop over the <code>lst</code> argument.</p>
|
||||
<p>Thus the total number of basic operations performed is <span class="math inline">\(n\)</span>, and so the running time is <span class="math inline">\(RT_{print\_items}(n) = n\)</span>, which is <span class="math inline">\(\Theta(n)\)</span>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p>Here is a second example, which has a similar structure to our first example, but also features slightly more code, using the familiar loop accumulator pattern.</p>
|
||||
<div class="example">
|
||||
<p>Analyse the running time of the following function.</p>
|
||||
<div class="sourceCode" id="cb3"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1"></a><span class="kw">def</span> my_sum(numbers: <span class="bu">list</span>[<span class="bu">int</span>]) <span class="op">-></span> <span class="bu">int</span>:</span>
|
||||
<span id="cb3-2"><a href="#cb3-2"></a> sum_so_far <span class="op">=</span> <span class="dv">0</span></span>
|
||||
<span id="cb3-3"><a href="#cb3-3"></a></span>
|
||||
<span id="cb3-4"><a href="#cb3-4"></a> <span class="cf">for</span> number <span class="kw">in</span> numbers:</span>
|
||||
<span id="cb3-5"><a href="#cb3-5"></a> sum_so_far <span class="op">=</span> sum_so_far <span class="op">+</span> number</span>
|
||||
<span id="cb3-6"><a href="#cb3-6"></a></span>
|
||||
<span id="cb3-7"><a href="#cb3-7"></a> <span class="cf">return</span> sum_so_far</span></code></pre></div>
|
||||
<div class="analysis">
|
||||
<p>Let <span class="math inline">\(n\)</span> be the length of the input list (i.e., <code>numbers</code>).</p>
|
||||
<p>This function body consists of three statements (with the middle statement, the for loop, itself containing more statements). To analyse the total running time of the function, we need to count each statement separately:</p>
|
||||
<ul>
|
||||
<li>The assignment statement <code>sum_so_far = 0</code> counts as 1 step, as its running time does not depend on the length of <code>numbers</code>.</li>
|
||||
<li>The for loop takes <span class="math inline">\(n\)</span> steps: it has <span class="math inline">\(n\)</span> iterations, and each iteration takes 1 step.<label for="sn-3" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-3" class="margin-toggle"/><span class="sidenote"> Remember that we’re treating all arithmetic operations as constant time here.</span></li>
|
||||
<li>The return statement counts as 1 step: it, too, has running time that does not depend on the length of <code>numbers</code>.</li>
|
||||
</ul>
|
||||
<p>The total running time is the sum of these three parts: <span class="math inline">\(1 + n + 1 = n + 2\)</span>, which is <span class="math inline">\(\Theta(n)\)</span>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<h2 id="nested-loops">Nested loops</h2>
|
||||
<p>It is quite possible to have nested loops in a function body, and analyze the running time in the same fashion. The simplest method of tackling such functions is to count the number of repeated basic operations in a loop starting with the <em>innermost</em> loop and working your way out.</p>
|
||||
<div class="example">
|
||||
<p>Consider the following function.</p>
|
||||
<div class="sourceCode" id="cb4"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb4-1"><a href="#cb4-1"></a><span class="kw">def</span> print_sums(lst: <span class="bu">list</span>) <span class="op">-></span> <span class="va">None</span>:</span>
|
||||
<span id="cb4-2"><a href="#cb4-2"></a> <span class="cf">for</span> item1 <span class="kw">in</span> lst:</span>
|
||||
<span id="cb4-3"><a href="#cb4-3"></a> <span class="cf">for</span> item2 <span class="kw">in</span> lst:</span>
|
||||
<span id="cb4-4"><a href="#cb4-4"></a> <span class="bu">print</span>(item1 <span class="op">+</span> item2)</span></code></pre></div>
|
||||
<p>Perform a runtime analysis of <code>print_sums</code>.</p>
|
||||
<div class="analysis">
|
||||
<p>Let <span class="math inline">\(n\)</span> be the length of <code>lst</code>.</p>
|
||||
<p>The inner loop (<code>for item2 in lst</code>) runs <span class="math inline">\(n\)</span> times (once per item in <code>lst</code>), and each iteration is just a single basic operation.</p>
|
||||
<p>But the entire inner loop is itself repeated, since it is inside another loop. The outer loop runs <span class="math inline">\(n\)</span> times as well, and each of its iterations takes <span class="math inline">\(n\)</span> operations.</p>
|
||||
<p>So then the total number of basic operations is <span class="math display">\[\begin{align*}
|
||||
RT_{print\_sums}(n)
|
||||
&= \text{steps for the inner loop} \times \text{number of times inner loop is repeated} \\
|
||||
&= n \times n \\
|
||||
&= n^2
|
||||
\end{align*}\]</span></p>
|
||||
<p>So the running time of this algorithm is <span class="math inline">\(\Theta(n^2)\)</span>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p>Students often make the mistake, however, that the number of nested loops should always be the exponent of <span class="math inline">\(n\)</span> in the Big-O expression.<label for="sn-4" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-4" class="margin-toggle"/><span class="sidenote">E.g., two levels of nested loops always becomes <span class="math inline">\(\Theta(n^2)\)</span>.</span> However, things are not that simple, and in particular, not every loop takes <span class="math inline">\(n\)</span> iterations.</p>
|
||||
<div class="example">
|
||||
<p>Consider the following function:</p>
|
||||
<div class="sourceCode" id="cb5"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb5-1"><a href="#cb5-1"></a><span class="kw">def</span> f(lst: <span class="bu">list</span>[<span class="bu">int</span>]) <span class="op">-></span> <span class="va">None</span>:</span>
|
||||
<span id="cb5-2"><a href="#cb5-2"></a> <span class="cf">for</span> item <span class="kw">in</span> lst:</span>
|
||||
<span id="cb5-3"><a href="#cb5-3"></a> <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, <span class="dv">10</span>):</span>
|
||||
<span id="cb5-4"><a href="#cb5-4"></a> <span class="bu">print</span>(item <span class="op">+</span> i)</span></code></pre></div>
|
||||
<p>Perform a runtime analysis of this function.</p>
|
||||
<div class="analysis">
|
||||
<p>Let <span class="math inline">\(n\)</span> be the length of the input list <code>lst</code>. The inner loop repeats 10 times, and each iteration is again a single basic operation, for a total of 10 basic operations. The outer loop repeats <span class="math inline">\(n\)</span> times, and each iteration takes 10 steps, for a total of <span class="math inline">\(10n\)</span> steps. So the running time of this function is <span class="math inline">\(\Theta(n)\)</span>. (Even though it has a nested loop!)</p>
|
||||
<p><em>Alternative, more concise analysis</em>. The inner loop’s running time doesn’t depend on the number of items in the input list, so we can count it as a single basic operation.</p>
|
||||
<p>The outer loop runs <span class="math inline">\(n\)</span> times, and each iteration takes <span class="math inline">\(1\)</span> step, for a total of <span class="math inline">\(n\)</span> steps, which is <span class="math inline">\(\Theta(n)\)</span>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p>When we are analyzing the running time of two blocks of code executed in sequence (one after the other), we add together their individual running times. The sum theorems are particularly helpful here, as it tells us that we can simply compute Theta expressions for the blocks individually, and then combine them just by taking the fastest-growing one. Because Theta expressions are a simplification of exact mathematical function expressions, taking this approach is often easier and faster than trying to count an exact number steps for the entire function.<label for="sn-5" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-5" class="margin-toggle"/><span class="sidenote">E.g., <span class="math inline">\(\Theta(n^2)\)</span> is simpler than <span class="math inline">\(10n^2 + 0.001n + 165\)</span>.</span></p>
|
||||
<div class="example">
|
||||
<p>Analyze the running time of the following function, which is a combination of two previous functions.</p>
|
||||
<div class="sourceCode" id="cb6"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb6-1"><a href="#cb6-1"></a><span class="kw">def</span> combined(lst: <span class="bu">list</span>[<span class="bu">int</span>]) <span class="op">-></span> <span class="va">None</span>:</span>
|
||||
<span id="cb6-2"><a href="#cb6-2"></a> <span class="co"># Loop 1</span></span>
|
||||
<span id="cb6-3"><a href="#cb6-3"></a> <span class="cf">for</span> item <span class="kw">in</span> lst:</span>
|
||||
<span id="cb6-4"><a href="#cb6-4"></a> <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">10</span>):</span>
|
||||
<span id="cb6-5"><a href="#cb6-5"></a> <span class="bu">print</span>(item <span class="op">+</span> i)</span>
|
||||
<span id="cb6-6"><a href="#cb6-6"></a></span>
|
||||
<span id="cb6-7"><a href="#cb6-7"></a> <span class="co"># Loop 2</span></span>
|
||||
<span id="cb6-8"><a href="#cb6-8"></a> <span class="cf">for</span> item1 <span class="kw">in</span> lst:</span>
|
||||
<span id="cb6-9"><a href="#cb6-9"></a> <span class="cf">for</span> item2 <span class="kw">in</span> lst:</span>
|
||||
<span id="cb6-10"><a href="#cb6-10"></a> <span class="bu">print</span>(item1 <span class="op">+</span> item2)</span></code></pre></div>
|
||||
<div class="analysis">
|
||||
<p>Let <span class="math inline">\(n\)</span> be the length of <code>lst</code>. We have already seen that the first loop runs in time <span class="math inline">\(\Theta(n)\)</span>, while the second loop runs in time <span class="math inline">\(\Theta(n^2)\)</span>.<label for="sn-6" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-6" class="margin-toggle"/><span class="sidenote">By “runs in time <span class="math inline">\(\Theta(n)\)</span>,” we mean that the number of basic operations of the second loop is a function <span class="math inline">\(f(n) \in \Theta(n)\)</span>.</span></p>
|
||||
<p>By the <em>Sum of Functions</em> theorem from the previous section, we can conclude that <code>combined</code> runs in time <span class="math inline">\(\Theta(n^2)\)</span>. (Since <span class="math inline">\(n \in \cO(n^2)\)</span>.)</p>
|
||||
</div>
|
||||
</div>
|
||||
<h3 id="loop-iterations-with-changing-costs">Loop iterations with changing costs</h3>
|
||||
<p>Now let’s look at one last example in this section, which is a function that prints out the sum of all distinct pairs of integers from a given list.</p>
|
||||
<div class="example">
|
||||
<p>Analyze the running time of the following function</p>
|
||||
<div class="sourceCode" id="cb7"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb7-1"><a href="#cb7-1"></a><span class="kw">def</span> all_pairs(lst: <span class="bu">list</span>[<span class="bu">int</span>]) <span class="op">-></span> <span class="va">None</span>:</span>
|
||||
<span id="cb7-2"><a href="#cb7-2"></a> <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, <span class="bu">len</span>(lst)):</span>
|
||||
<span id="cb7-3"><a href="#cb7-3"></a> <span class="cf">for</span> j <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, i):</span>
|
||||
<span id="cb7-4"><a href="#cb7-4"></a> <span class="bu">print</span>(lst[i] <span class="op">+</span> lst[j])</span></code></pre></div>
|
||||
<div class="discussion">
|
||||
<p>Like previous examples, this function has a nested loop. However, unlike those examples, here the inner loop’s running time depends on the current value of <code>i</code>, i.e., which iteration of the outer loop we’re on.</p>
|
||||
<p>This means we cannot take the previous approach of calculating the cost of the inner loop, and multiplying it by the number of iterations of the outer loop; this only works if the cost of each outer loop iteration is the same.</p>
|
||||
<p>So instead, we need to manually add up the cost of each iteration of the outer loop, which depends on the number of iterations of the inner loop. More specifically, since <span class="math inline">\(j\)</span> goes from <span class="math inline">\(0\)</span> to <span class="math inline">\(i-1\)</span>, the number of iterations of the inner loop is <span class="math inline">\(i\)</span>, and each iteration of the inner loop counts as one basic operation.</p>
|
||||
<p>Let’s see how to do this in a formal analysis.</p>
|
||||
</div>
|
||||
<div class="analysis">
|
||||
<p>Let <span class="math inline">\(n\)</span> be the length of the input list.</p>
|
||||
<p>We start by analysing the running time of the inner loop for a <em>fixed</em> iteration of the outer loop, and a fixed value of <span class="math inline">\(i\)</span>.</p>
|
||||
<ul>
|
||||
<li>The inner loop iterates <span class="math inline">\(i\)</span> times (for <span class="math inline">\(j\)</span> going from 0 to <span class="math inline">\(i - 1\)</span>), and each iteration takes one step (constant time).<label for="sn-7" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-7" class="margin-toggle"/><span class="sidenote">Here, list indexing is counted as constant time—we’ll explore this more a bit later this chapter.</span> Therefore the cost of the inner loop is <span class="math inline">\(i\)</span> steps, for one iteration of the outer loop.</li>
|
||||
</ul>
|
||||
<p>Now, the outer loop iterates <span class="math inline">\(n\)</span> times for <span class="math inline">\(i\)</span> going from 0 to <span class="math inline">\(n - 1\)</span>. But here the cost of each iteration is not constant. Instead, the cost of iteration <span class="math inline">\(i\)</span> is <span class="math inline">\(i\)</span> steps, and so the total cost of the outer loop is:</p>
|
||||
<p><span class="math display">\[\sum_{i=0}^{n-1} i = \frac{n(n - 1)}{2}\]</span></p>
|
||||
<p>Here we used the summation formula for the sum of the first <span class="math inline">\(n\)</span> natural numbers, which is reviewed in <a href="../C-math-reference/01-summations-products.html">Appendix C.1</a>.</p>
|
||||
<p>And so the total number of steps taken by <code>all_pairs</code> is <span class="math inline">\(\frac{n(n - 1)}{2}\)</span>, which is <span class="math inline">\(\Theta(n^2)\)</span>.<label for="sn-8" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-8" class="margin-toggle"/><span class="sidenote"> Note that we can write <span class="math inline">\(\frac{n(n - 1)}{2} = \frac{1}{2} n^2 - \frac{1}{2} n\)</span>.</span></p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
<footer>
|
||||
<a href="https://www.teach.cs.toronto.edu/~csc110y/fall/notes/">CSC110 Course Notes Home</a>
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,322 @@
|
||||
<!DOCTYPE html>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="generator" content="pandoc" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||||
<title>8.5 Analyzing Comprehensions and While Loops</title>
|
||||
<style>
|
||||
code{white-space: pre-wrap;}
|
||||
span.smallcaps{font-variant: small-caps;}
|
||||
span.underline{text-decoration: underline;}
|
||||
div.column{display: inline-block; vertical-align: top; width: 50%;}
|
||||
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||||
ul.task-list{list-style: none;}
|
||||
pre > code.sourceCode { white-space: pre; position: relative; }
|
||||
pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
|
||||
pre > code.sourceCode > span:empty { height: 1.2em; }
|
||||
code.sourceCode > span { color: inherit; text-decoration: inherit; }
|
||||
div.sourceCode { margin: 1em 0; }
|
||||
pre.sourceCode { margin: 0; }
|
||||
@media screen {
|
||||
div.sourceCode { overflow: auto; }
|
||||
}
|
||||
@media print {
|
||||
pre > code.sourceCode { white-space: pre-wrap; }
|
||||
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
|
||||
}
|
||||
pre.numberSource code
|
||||
{ counter-reset: source-line 0; }
|
||||
pre.numberSource code > span
|
||||
{ position: relative; left: -4em; counter-increment: source-line; }
|
||||
pre.numberSource code > span > a:first-child::before
|
||||
{ content: counter(source-line);
|
||||
position: relative; left: -1em; text-align: right; vertical-align: baseline;
|
||||
border: none; display: inline-block;
|
||||
-webkit-touch-callout: none; -webkit-user-select: none;
|
||||
-khtml-user-select: none; -moz-user-select: none;
|
||||
-ms-user-select: none; user-select: none;
|
||||
padding: 0 4px; width: 4em;
|
||||
color: #aaaaaa;
|
||||
}
|
||||
pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; }
|
||||
div.sourceCode
|
||||
{ }
|
||||
@media screen {
|
||||
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
|
||||
}
|
||||
code span.al { color: #ff0000; font-weight: bold; } /* Alert */
|
||||
code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
|
||||
code span.at { color: #7d9029; } /* Attribute */
|
||||
code span.bn { color: #40a070; } /* BaseN */
|
||||
code span.bu { } /* BuiltIn */
|
||||
code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
|
||||
code span.ch { color: #4070a0; } /* Char */
|
||||
code span.cn { color: #880000; } /* Constant */
|
||||
code span.co { color: #60a0b0; font-style: italic; } /* Comment */
|
||||
code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
|
||||
code span.do { color: #ba2121; font-style: italic; } /* Documentation */
|
||||
code span.dt { color: #902000; } /* DataType */
|
||||
code span.dv { color: #40a070; } /* DecVal */
|
||||
code span.er { color: #ff0000; font-weight: bold; } /* Error */
|
||||
code span.ex { } /* Extension */
|
||||
code span.fl { color: #40a070; } /* Float */
|
||||
code span.fu { color: #06287e; } /* Function */
|
||||
code span.im { } /* Import */
|
||||
code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
|
||||
code span.kw { color: #007020; font-weight: bold; } /* Keyword */
|
||||
code span.op { color: #666666; } /* Operator */
|
||||
code span.ot { color: #007020; } /* Other */
|
||||
code span.pp { color: #bc7a00; } /* Preprocessor */
|
||||
code span.sc { color: #4070a0; } /* SpecialChar */
|
||||
code span.ss { color: #bb6688; } /* SpecialString */
|
||||
code span.st { color: #4070a0; } /* String */
|
||||
code span.va { color: #19177c; } /* Variable */
|
||||
code span.vs { color: #4070a0; } /* VerbatimString */
|
||||
code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
|
||||
</style>
|
||||
<link rel="stylesheet" href="../tufte.css" />
|
||||
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js" type="text/javascript"></script>
|
||||
<!--[if lt IE 9]>
|
||||
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div style="display:none">
|
||||
\(
|
||||
\newcommand{\NOT}{\neg}
|
||||
\newcommand{\AND}{\wedge}
|
||||
\newcommand{\OR}{\vee}
|
||||
\newcommand{\XOR}{\oplus}
|
||||
\newcommand{\IMP}{\Rightarrow}
|
||||
\newcommand{\IFF}{\Leftrightarrow}
|
||||
\newcommand{\TRUE}{\text{True}\xspace}
|
||||
\newcommand{\FALSE}{\text{False}\xspace}
|
||||
\newcommand{\IN}{\,{\in}\,}
|
||||
\newcommand{\NOTIN}{\,{\notin}\,}
|
||||
\newcommand{\TO}{\rightarrow}
|
||||
\newcommand{\DIV}{\mid}
|
||||
\newcommand{\NDIV}{\nmid}
|
||||
\newcommand{\MOD}[1]{\pmod{#1}}
|
||||
\newcommand{\MODS}[1]{\ (\text{mod}\ #1)}
|
||||
\newcommand{\N}{\mathbb N}
|
||||
\newcommand{\Z}{\mathbb Z}
|
||||
\newcommand{\Q}{\mathbb Q}
|
||||
\newcommand{\R}{\mathbb R}
|
||||
\newcommand{\C}{\mathbb C}
|
||||
\newcommand{\cA}{\mathcal A}
|
||||
\newcommand{\cB}{\mathcal B}
|
||||
\newcommand{\cC}{\mathcal C}
|
||||
\newcommand{\cD}{\mathcal D}
|
||||
\newcommand{\cE}{\mathcal E}
|
||||
\newcommand{\cF}{\mathcal F}
|
||||
\newcommand{\cG}{\mathcal G}
|
||||
\newcommand{\cH}{\mathcal H}
|
||||
\newcommand{\cI}{\mathcal I}
|
||||
\newcommand{\cJ}{\mathcal J}
|
||||
\newcommand{\cL}{\mathcal L}
|
||||
\newcommand{\cK}{\mathcal K}
|
||||
\newcommand{\cN}{\mathcal N}
|
||||
\newcommand{\cO}{\mathcal O}
|
||||
\newcommand{\cP}{\mathcal P}
|
||||
\newcommand{\cQ}{\mathcal Q}
|
||||
\newcommand{\cS}{\mathcal S}
|
||||
\newcommand{\cT}{\mathcal T}
|
||||
\newcommand{\cV}{\mathcal V}
|
||||
\newcommand{\cW}{\mathcal W}
|
||||
\newcommand{\cZ}{\mathcal Z}
|
||||
\newcommand{\emp}{\emptyset}
|
||||
\newcommand{\bs}{\backslash}
|
||||
\newcommand{\floor}[1]{\left \lfloor #1 \right \rfloor}
|
||||
\newcommand{\ceil}[1]{\left \lceil #1 \right \rceil}
|
||||
\newcommand{\abs}[1]{\left | #1 \right |}
|
||||
\newcommand{\xspace}{}
|
||||
\newcommand{\proofheader}[1]{\underline{\textbf{#1}}}
|
||||
\)
|
||||
</div>
|
||||
<header id="title-block-header">
|
||||
<h1 class="title">8.5 Analyzing Comprehensions and While Loops</h1>
|
||||
</header>
|
||||
<section>
|
||||
<p>In the previous section, we began our study of algorithm running time analysis by looking at functions that are implemented using for loops. We chose for loops as a starting point because they make explicit the <em>repeated statements</em> that occur when we execute a function body, while also being relatively straightforward to analyze because of their predicable iteration patterns.</p>
|
||||
<p>In this section, we’ll extend what we learned about for loops to two different kinds of Python code: comprehension expressions and while loop. We’ll see how all three obey similar patterns when it comes to repeating code, but while loops offer both more flexibility and more complexity in what they can do.</p>
|
||||
<h2 id="comprehensions">Comprehensions</h2>
|
||||
<p>Consider the following function:</p>
|
||||
<div class="sourceCode" id="cb1"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1"></a><span class="kw">def</span> square_all(numbers: <span class="bu">list</span>[<span class="bu">int</span>]) <span class="op">-></span> <span class="bu">list</span>[<span class="bu">int</span>]:</span>
|
||||
<span id="cb1-2"><a href="#cb1-2"></a> <span class="co">"""Return a new list containing the squares of the given numbers."""</span></span>
|
||||
<span id="cb1-3"><a href="#cb1-3"></a> <span class="cf">return</span> [x <span class="op">**</span> <span class="dv">2</span> <span class="cf">for</span> x <span class="kw">in</span> numbers]</span></code></pre></div>
|
||||
<div class="analysis">
|
||||
<p>How do we analyze the running time of this code? It turns out that we do so in the same way as a for loop:</p>
|
||||
<ol type="1">
|
||||
<li>We determine the number of steps required to evaluate the leftmost expression in the comprehension. In this case, evaluating <code>x ** 2</code> takes 1 step (i.e., is constant time).</li>
|
||||
<li>The collection that acts as the source of the comprehension (in our example, <code>numbers</code>), determines how many times the leftmost expression is evaluated.</li>
|
||||
</ol>
|
||||
<p>So let <span class="math inline">\(n\)</span> be the length of the input list <code>numbers</code>. The comprehension expression takes <span class="math inline">\(n\)</span> steps (1 step per element of <code>numbers</code>). So the running time of <code>square_all</code> is <span class="math inline">\(n\)</span> steps, which is <span class="math inline">\(\Theta(n)\)</span>.</p>
|
||||
</div>
|
||||
<p>Importantly, the fact that a comprehension is creating a new collection (in our above example, a list) does <em>not</em> count as additional time when analysing the cost of a comprehension. This is true for all three of list, set, and dictionary comprehensions, and so the same analysis would hold in the above function if we had used a set or dictionary comprehension instead.</p>
|
||||
<h2 id="while-loops">While loops</h2>
|
||||
<p>Analysing the running time of code involving while loops follows the same principle using for loops: we calculate the sum of the different loop iterations, either using multiplication (when the iteration running time is constant) or a summation (when the iterations have different running times). There is one subtle twist, though: a while loop requires that we write statements to initialize the loop variable(s) before the loop, and update the loop variable(s) inside the loop body. We must be careful to count the cost of these statements as well, just like we did for statements involving loop accumulators in the previous section.</p>
|
||||
<p>To keep things simple, our first example is a simple rewriting of an earlier example using a while loop instead of a for loop.</p>
|
||||
<div class="example">
|
||||
<p>Analyse the running time of the following function.</p>
|
||||
<div class="sourceCode" id="cb2"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1"></a><span class="kw">def</span> my_sum_v2(numbers: <span class="bu">list</span>[<span class="bu">int</span>]) <span class="op">-></span> <span class="bu">int</span>:</span>
|
||||
<span id="cb2-2"><a href="#cb2-2"></a> <span class="co">"""Return the sum of the given numbers."""</span></span>
|
||||
<span id="cb2-3"><a href="#cb2-3"></a> sum_so_far <span class="op">=</span> <span class="dv">0</span></span>
|
||||
<span id="cb2-4"><a href="#cb2-4"></a> i <span class="op">=</span> <span class="dv">0</span></span>
|
||||
<span id="cb2-5"><a href="#cb2-5"></a></span>
|
||||
<span id="cb2-6"><a href="#cb2-6"></a> <span class="cf">while</span> i <span class="op"><</span> <span class="bu">len</span>(numbers):</span>
|
||||
<span id="cb2-7"><a href="#cb2-7"></a> sum_so_far <span class="op">=</span> sum_so_far <span class="op">+</span> numbers[i]</span>
|
||||
<span id="cb2-8"><a href="#cb2-8"></a> i <span class="op">=</span> i <span class="op">+</span> <span class="dv">1</span></span>
|
||||
<span id="cb2-9"><a href="#cb2-9"></a></span>
|
||||
<span id="cb2-10"><a href="#cb2-10"></a> <span class="cf">return</span> sum_so_far</span></code></pre></div>
|
||||
<div class="analysis">
|
||||
<p>Let <span class="math inline">\(n\)</span> be the length of the input <code>numbers</code>.</p>
|
||||
<p>In this function, we now have both an accumulator and the loop variable to worry about. We can still divide up the function into three parts, and compute the cost of each part separately.</p>
|
||||
<ol type="1">
|
||||
<li><p>The cost of the assignment statements <code>sum_so_far = 0</code> and <code>i = 0</code> is constant time. We’ll count this as a constant-time block of code, which is just <em>1</em> step.<label for="sn-0" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-0" class="margin-toggle"/><span class="sidenote"> This might be a bit surprising, because they are two lines of code and look like two separate “actions”. The power of our asymptotic notation is that whether we count this block of code as 1 step or 2, we get the same Theta bound in the end! And so we just go with the simpler one here, but you’re welcome to count this as “two steps” in your own analyses if you find that more intuitive.</span></p></li>
|
||||
<li><p>To analyse the while loop, we need to determine the cost of each iteration and the total number of iterations, just like a for loop.</p>
|
||||
<ul>
|
||||
<li>Each iteration is constant time, so we’ll count that as one step.</li>
|
||||
<li>There are <span class="math inline">\(n\)</span> iterations, since <code>i</code> starts at 0 and increases by 1 until it reaches <span class="math inline">\(n\)</span>. Note that this is less obvious than the for loop version! Here we need to look at three different places in the code: how <code>i</code> is initialized, how <code>i</code> is updated inside the loop body, and how <code>i</code> is used in the loop condition.</li>
|
||||
</ul></li>
|
||||
<li><p>The return statement again takes constant time, and so counts as 1 step.</p></li>
|
||||
</ol>
|
||||
<p>So the total running time is <span class="math inline">\(1 + n + 1 = n + 2\)</span>, which is <span class="math inline">\(\Theta(n)\)</span>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p>Now, the previous example was a little contrived because we could have implemented the same function more simply using a for loop. Here is another example, which uses a while loop to compute powers of two to act as indexes into a list.</p>
|
||||
<div class="example">
|
||||
<p>Analyse the running time of the following function.</p>
|
||||
<div class="sourceCode" id="cb3"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1"></a><span class="kw">def</span> my_sum_powers_of_two(numbers: <span class="bu">list</span>[<span class="bu">int</span>]) <span class="op">-></span> <span class="bu">int</span>:</span>
|
||||
<span id="cb3-2"><a href="#cb3-2"></a> <span class="co">"""Return the sum of the given numbers whose indexes are powers of 2.</span></span>
|
||||
<span id="cb3-3"><a href="#cb3-3"></a></span>
|
||||
<span id="cb3-4"><a href="#cb3-4"></a><span class="co"> That is, return numbers[1] + numbers[2] + numbers[4] + numbers[8] + ...</span></span>
|
||||
<span id="cb3-5"><a href="#cb3-5"></a><span class="co"> """</span></span>
|
||||
<span id="cb3-6"><a href="#cb3-6"></a> sum_so_far <span class="op">=</span> <span class="dv">0</span></span>
|
||||
<span id="cb3-7"><a href="#cb3-7"></a> i <span class="op">=</span> <span class="dv">1</span></span>
|
||||
<span id="cb3-8"><a href="#cb3-8"></a></span>
|
||||
<span id="cb3-9"><a href="#cb3-9"></a> <span class="cf">while</span> i <span class="op"><</span> <span class="bu">len</span>(numbers):</span>
|
||||
<span id="cb3-10"><a href="#cb3-10"></a> sum_so_far <span class="op">=</span> sum_so_far <span class="op">+</span> numbers[i]</span>
|
||||
<span id="cb3-11"><a href="#cb3-11"></a> i <span class="op">=</span> i <span class="op">*</span> <span class="dv">2</span></span>
|
||||
<span id="cb3-12"><a href="#cb3-12"></a></span>
|
||||
<span id="cb3-13"><a href="#cb3-13"></a> <span class="cf">return</span> sum_so_far</span></code></pre></div>
|
||||
<div class="analysis">
|
||||
<p>Let <span class="math inline">\(n\)</span> be the length of the input list <code>numbers</code>.</p>
|
||||
<p>This code has much of the same structure as <code>my_sum_v2</code>, and we can reuse most of the same analysis here. In particular, we’ll still count the initial assignment statements as 1 step, and the return statement as 1 step. To analyse the loop, we still need the number of steps per iteration and the total number of iterations. Each iteration still takes constant time (1 step), same as <code>my_sum_v2</code>. It is the number of loop iterations that is most challenging.</p>
|
||||
<p>To determine the number of loop iterations, we need to take into account the initial value of <code>i</code>, how <code>i</code> is updated, and how <code>i</code> is used in the while loop condition. More formally, we follow these steps:</p>
|
||||
<ol type="1">
|
||||
<li><p>Find a pattern for how <code>i</code> changes at each loop iteration, and a general formula formula for <span class="math inline">\(i_k\)</span>, the value of <code>i</code> after <span class="math inline">\(k\)</span> iterations. For relatively simple updates, we can find a pattern by writing a small loop tracing table, showing the value of the loop variable at the <em>end</em> of the iteration.</p>
|
||||
<div class="reference-table">
|
||||
<table>
|
||||
<thead>
|
||||
<tr class="header">
|
||||
<th style="text-align: left;">Iteration</th>
|
||||
<th>Value of <code>i</code></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr class="odd">
|
||||
<td style="text-align: left;">0</td>
|
||||
<td>1</td>
|
||||
</tr>
|
||||
<tr class="even">
|
||||
<td style="text-align: left;">1</td>
|
||||
<td>2</td>
|
||||
</tr>
|
||||
<tr class="odd">
|
||||
<td style="text-align: left;">2</td>
|
||||
<td>4</td>
|
||||
</tr>
|
||||
<tr class="even">
|
||||
<td style="text-align: left;">3</td>
|
||||
<td>8</td>
|
||||
</tr>
|
||||
<tr class="odd">
|
||||
<td style="text-align: left;">4</td>
|
||||
<td>16</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<p>So we find that after <span class="math inline">\(k\)</span> iterations, <span class="math inline">\(i_k = 2^k\)</span>.<label for="sn-1" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-1" class="margin-toggle"/><span class="sidenote"> Note that we haven’t <em>proved</em> that this formula is true; a formal proof would require a proof by induction, which you may have already seen in your math classes.</span></p></li>
|
||||
<li><p>We know the while loop continues while <code>i < len(numbers)</code>. Another way to phrase this is that the while loop continues <em>until</em> <code>i >= len(numbers)</code>.</p>
|
||||
<p>So to find the number of iterations, we need to find the smallest value of <span class="math inline">\(k\)</span> such that <span class="math inline">\(i_k \geq n\)</span> (making the loop condition False). This is where our formula for <span class="math inline">\(i_k\)</span> comes in:</p>
|
||||
<p><span class="math display">\[\begin{align*}
|
||||
i_k &\geq n \\
|
||||
2^k &\geq n \\
|
||||
k &\geq \log_2 n
|
||||
\end{align*}\]</span> So we need to find the smallest value of <span class="math inline">\(k\)</span> such that <span class="math inline">\(k \geq \log_2 n\)</span>. This is exactly the definition of the ceiling function, and so the smallest value of <span class="math inline">\(k\)</span> is <span class="math inline">\(\ceil{\log_2 n}\)</span>.</p></li>
|
||||
</ol>
|
||||
<p>So the while loop iterates <span class="math inline">\(\ceil{\log_2 n}\)</span> times, with 1 step per iteration, for a total of <span class="math inline">\(\ceil{\log_2 n}\)</span> steps.</p>
|
||||
<p>Putting it all together, the function <code>my_sum_powers_of_two</code> has a running time of <span class="math inline">\(1 + \ceil{\log_2 n} + 1 = \ceil{\log_2 n} + 2\)</span>, which is <span class="math inline">\(\Theta(\log n)\)</span>.<label for="sn-2" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-2" class="margin-toggle"/><span class="sidenote"> Note that our convention is to drop the base of the log when writing a Theta expression, since all bases <span class="math inline">\(> 1\)</span> are equivalent to each other in Theta bounds.</span></p>
|
||||
</div>
|
||||
</div>
|
||||
<h2 id="a-trickier-example">A trickier example</h2>
|
||||
<p>It turns out that the extreme flexibility of while loops can make analysing their running time much more subtle than it might appear. Our next example considers a standard loop, with a twist in how the loop variable changes at each iteration.</p>
|
||||
<div class="sourceCode" id="cb4"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb4-1"><a href="#cb4-1"></a><span class="kw">def</span> twisty(n: <span class="bu">int</span>) <span class="op">-></span> <span class="bu">int</span>:</span>
|
||||
<span id="cb4-2"><a href="#cb4-2"></a> <span class="co">"""Return the number of iterations it takes for this special loop to stop</span></span>
|
||||
<span id="cb4-3"><a href="#cb4-3"></a><span class="co"> for the given n.</span></span>
|
||||
<span id="cb4-4"><a href="#cb4-4"></a><span class="co"> """</span></span>
|
||||
<span id="cb4-5"><a href="#cb4-5"></a> iterations_so_far <span class="op">=</span> <span class="dv">0</span></span>
|
||||
<span id="cb4-6"><a href="#cb4-6"></a> x <span class="op">=</span> n</span>
|
||||
<span id="cb4-7"><a href="#cb4-7"></a> <span class="cf">while</span> x <span class="op">></span> <span class="dv">1</span>:</span>
|
||||
<span id="cb4-8"><a href="#cb4-8"></a> <span class="cf">if</span> x <span class="op">%</span> <span class="dv">2</span> <span class="op">==</span> <span class="dv">0</span>:</span>
|
||||
<span id="cb4-9"><a href="#cb4-9"></a> x <span class="op">=</span> x <span class="op">/</span> <span class="dv">2</span></span>
|
||||
<span id="cb4-10"><a href="#cb4-10"></a> <span class="cf">else</span>:</span>
|
||||
<span id="cb4-11"><a href="#cb4-11"></a> x <span class="op">=</span> <span class="dv">2</span> <span class="op">*</span> x <span class="op">-</span> <span class="dv">2</span></span>
|
||||
<span id="cb4-12"><a href="#cb4-12"></a> iterations_so_far <span class="op">=</span> iterations_so_far <span class="op">+</span> <span class="dv">1</span></span>
|
||||
<span id="cb4-13"><a href="#cb4-13"></a></span>
|
||||
<span id="cb4-14"><a href="#cb4-14"></a> <span class="cf">return</span> iterations_so_far</span></code></pre></div>
|
||||
<p>Even though the individual lines of code in this example are simple, they combine to form a pretty complex situation. The challenge with analyzing the runtime of this function is that, unlike previous examples, here the loop variable <code>x</code> does not always get closer to the loop stopping condition; sometimes it does (when divided by two), and sometimes it increases!</p>
|
||||
<p>The key insight into analyzing the runtime of this function is that we don’t just need to look at what happens after a single loop iteration, but instead perform a more sophisticated analysis based on <em>multiple</em> iterations.<label for="sn-3" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-3" class="margin-toggle"/><span class="sidenote"> As preparation, try tracing <code>twisty</code> on inputs <span class="math inline">\(7\)</span>, <span class="math inline">\(9\)</span>, and <span class="math inline">\(11\)</span>.</span> More concretely, we’ll prove the following claim.</p>
|
||||
<div class="claim">
|
||||
<p>For any integer value of <code>x</code> greater than <span class="math inline">\(2\)</span>, after <em>two</em> iterations of the loop in <code>twisty</code> the value of <code>x</code> decreases by at least one.</p>
|
||||
<div class="proof">
|
||||
<p>Let <span class="math inline">\(x_0\)</span> be the value of variable <code>x</code> at some iteration of the loop, and assume <span class="math inline">\(x_0 > 2\)</span>. Let <span class="math inline">\(x_1\)</span> be the value of <span class="math inline">\(x\)</span> after one loop iteration, and <span class="math inline">\(x_2\)</span> the value of <span class="math inline">\(x\)</span> after two loop iterations. We want to prove that <span class="math inline">\(x_2 \leq x_0 - 1\)</span>.</p>
|
||||
<p>We divide up this proof into four cases, based on the remainder of <span class="math inline">\(x_0\)</span> when dividing by four.<label for="sn-4" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-4" class="margin-toggle"/><span class="sidenote">The intuition for these cases is that this determines whether <span class="math inline">\(x_0\)</span> is even/odd, and whether <span class="math inline">\(x_1\)</span> is even/odd.</span> We’ll only do two cases here to illustrate the main idea, and leave the last two cases as an exercise.</p>
|
||||
<p><strong>Case 1</strong>: Assume <span class="math inline">\(4 \DIV x_0\)</span>, i.e., <span class="math inline">\(\exists k \in \Z,~ x_0 = 4k\)</span>.</p>
|
||||
<p>In this case, <span class="math inline">\(x_0\)</span> is even, so the <code>if</code> branch executes in the first loop iteration, and so <span class="math inline">\(x_1 = \frac{x_0}{2} = 2k\)</span>. And so then <span class="math inline">\(x_1\)</span> is also even, and so the <code>if</code> branch executes again: <span class="math inline">\(x_2 = \frac{x_1}{2} = k\)</span>.</p>
|
||||
<p>So then <span class="math inline">\(x_2 = \frac{1}{4}x_0 \leq x_0 - 1\)</span> (since <span class="math inline">\(x_0 \geq 4\)</span>), as required.</p>
|
||||
<p><strong>Case 2</strong>: Assume <span class="math inline">\(4 \DIV x_0 - 1\)</span>, i.e., <span class="math inline">\(\exists k \in \Z,~ x_0 = 4k + 1\)</span>.</p>
|
||||
<p>In this case, <span class="math inline">\(x_0\)</span> is odd, so the <code>else</code> branch executes in the first loop iteration, and so <span class="math inline">\(x_1 = 2x_0 - 2 = 8k\)</span>. Then <span class="math inline">\(x_1\)</span> is even, and so <span class="math inline">\(x_2 = \frac{x_1}{2} = 4k\)</span>.</p>
|
||||
<p>So then <span class="math inline">\(x_2 = 4k = x_0 - 1\)</span>, as required.</p>
|
||||
<p><strong>Cases 3 and 4</strong>: left as exercises.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p>Now let’s see how take this claim and use it to formally analyse the running time of <code>twisty</code>.</p>
|
||||
<div class="analysis">
|
||||
<p>(<em>Analysis of <code>twisty</code></em>)</p>
|
||||
<p>As before, we count the variable initializations before the while loop as 1 step, and the return statement as 1 step.</p>
|
||||
<p>For the while loop:</p>
|
||||
<ul>
|
||||
<li><p>The loop body also takes 1 step, since all of the code consists of operations that do not depend on the size of the input <span class="math inline">\(n\)</span>.</p></li>
|
||||
<li><p>To count the number of loop iterations, we first observe that <span class="math inline">\(x\)</span> starts at <span class="math inline">\(n\)</span> and the loop terminates when <span class="math inline">\(x\)</span> reaches 1 or less. The <em>Claim</em> tells us that after every two iterations, the value of <span class="math inline">\(x\)</span> decreases by at least one.</p>
|
||||
<p>So then after 2 iterations, <span class="math inline">\(x \leq n - 1\)</span>, after 4 iterations, <span class="math inline">\(x \leq n - 2\)</span>, and in general, after <span class="math inline">\(2k\)</span> iterations, <span class="math inline">\(x \leq n - k\)</span>. This tells us that after <span class="math inline">\(2(n - 1)\)</span> loop iterations, <span class="math inline">\(x \leq n - (n - 1) = 1\)</span>, and so the loop must stop.</p></li>
|
||||
</ul>
|
||||
<p>This analysis tells us that the loop iterations <em>at most</em> <span class="math inline">\(2(n - 1)\)</span> times, and so takes <em>at most</em> <span class="math inline">\(2(n - 1)\)</span> steps (remember that each iteration takes 1 step).</p>
|
||||
<p>So the total running time of <code>twisty</code> is <em>at most</em> <span class="math inline">\(1 + 2(n - 1) + 1 = 2n\)</span> steps, which is <span class="math inline">\(\cO(n)\)</span>.</p>
|
||||
</div>
|
||||
<p>Something funny happened at the end of the above analysis: we did not actually compute the exact number of steps the function <code>twisty</code> takes, only an <em>upper bound</em> on the number of steps (signalled by our use of the phrase “at most”). This means that we were only able to conclude a Big-O bound, and not a Theta bound, on the running time of this function: its running time is <em>at most</em> <span class="math inline">\(\cO(n)\)</span>, but we don’t know whether this bound is tight.</p>
|
||||
<p>In fact, it isn’t! It is possible to prove something pretty remarkable about what happens to the variable <code>x</code> after <em>three</em> iterations of the twisty loop.</p>
|
||||
<div class="claim">
|
||||
<p>(<em>Improved claim</em>)</p>
|
||||
<p>For any integer value of <code>x</code> greater than <span class="math inline">\(2\)</span>, let <span class="math inline">\(x_0\)</span> be the initial value of <code>x</code> and let <span class="math inline">\(x_3\)</span> be the value of <code>x</code> after <em>three</em> loop iterations. Then <span class="math inline">\(\frac{1}{8} x_0 \leq x_3 \leq \frac{1}{2} x_0\)</span>.</p>
|
||||
</div>
|
||||
<p>It is a good exercise to prove this claim<label for="sn-5" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-5" class="margin-toggle"/><span class="sidenote"> Hint: you can use the same approach as the previous claim, but consider remainders when you divide by 8 instead of 4.</span> and then use this claim to conduct a more detailed running time analysis of <code>twisty</code>. When you do so, you should be able to show that the running time of <code>twisty</code> is both <span class="math inline">\(\cO(\log n)\)</span> and <span class="math inline">\(\Omega(\log n)\)</span>, and hence conclude that its running time is actually <span class="math inline">\(\Theta(\log n)\)</span>, not just <span class="math inline">\(\cO(n)\)</span>!</p>
|
||||
<!-- <div exercise>
|
||||
<div questions data-series="chapter5">
|
||||
\item
|
||||
The analysis we performed in the previous example is incomplete for a few reasons; our goal with this set of exercises is to complete it here.
|
||||
|
||||
(a) Complete the last two cases in the proof of the claim.
|
||||
(b) State and prove an analogous statement for how much $x$ must decrease by after *three* loop iterations.
|
||||
(c) Find an exact upper bound on the number of loop iterations taken by this algorithm. Your upper bound should be smaller (and therefore more accurate) than the one given in the example.
|
||||
(d) Finally, find, with proof, a good *lower bound* on the number of loop iterations taken by this algorithm.
|
||||
</div>
|
||||
</div> -->
|
||||
</section>
|
||||
<footer>
|
||||
<a href="https://www.teach.cs.toronto.edu/~csc110y/fall/notes/">CSC110 Course Notes Home</a>
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,367 @@
|
||||
<!DOCTYPE html>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="generator" content="pandoc" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||||
<title>8.6 Analyzing Built-In Data Type Operations</title>
|
||||
<style>
|
||||
code{white-space: pre-wrap;}
|
||||
span.smallcaps{font-variant: small-caps;}
|
||||
span.underline{text-decoration: underline;}
|
||||
div.column{display: inline-block; vertical-align: top; width: 50%;}
|
||||
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||||
ul.task-list{list-style: none;}
|
||||
pre > code.sourceCode { white-space: pre; position: relative; }
|
||||
pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
|
||||
pre > code.sourceCode > span:empty { height: 1.2em; }
|
||||
code.sourceCode > span { color: inherit; text-decoration: inherit; }
|
||||
div.sourceCode { margin: 1em 0; }
|
||||
pre.sourceCode { margin: 0; }
|
||||
@media screen {
|
||||
div.sourceCode { overflow: auto; }
|
||||
}
|
||||
@media print {
|
||||
pre > code.sourceCode { white-space: pre-wrap; }
|
||||
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
|
||||
}
|
||||
pre.numberSource code
|
||||
{ counter-reset: source-line 0; }
|
||||
pre.numberSource code > span
|
||||
{ position: relative; left: -4em; counter-increment: source-line; }
|
||||
pre.numberSource code > span > a:first-child::before
|
||||
{ content: counter(source-line);
|
||||
position: relative; left: -1em; text-align: right; vertical-align: baseline;
|
||||
border: none; display: inline-block;
|
||||
-webkit-touch-callout: none; -webkit-user-select: none;
|
||||
-khtml-user-select: none; -moz-user-select: none;
|
||||
-ms-user-select: none; user-select: none;
|
||||
padding: 0 4px; width: 4em;
|
||||
color: #aaaaaa;
|
||||
}
|
||||
pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; }
|
||||
div.sourceCode
|
||||
{ }
|
||||
@media screen {
|
||||
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
|
||||
}
|
||||
code span.al { color: #ff0000; font-weight: bold; } /* Alert */
|
||||
code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
|
||||
code span.at { color: #7d9029; } /* Attribute */
|
||||
code span.bn { color: #40a070; } /* BaseN */
|
||||
code span.bu { } /* BuiltIn */
|
||||
code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
|
||||
code span.ch { color: #4070a0; } /* Char */
|
||||
code span.cn { color: #880000; } /* Constant */
|
||||
code span.co { color: #60a0b0; font-style: italic; } /* Comment */
|
||||
code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
|
||||
code span.do { color: #ba2121; font-style: italic; } /* Documentation */
|
||||
code span.dt { color: #902000; } /* DataType */
|
||||
code span.dv { color: #40a070; } /* DecVal */
|
||||
code span.er { color: #ff0000; font-weight: bold; } /* Error */
|
||||
code span.ex { } /* Extension */
|
||||
code span.fl { color: #40a070; } /* Float */
|
||||
code span.fu { color: #06287e; } /* Function */
|
||||
code span.im { } /* Import */
|
||||
code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
|
||||
code span.kw { color: #007020; font-weight: bold; } /* Keyword */
|
||||
code span.op { color: #666666; } /* Operator */
|
||||
code span.ot { color: #007020; } /* Other */
|
||||
code span.pp { color: #bc7a00; } /* Preprocessor */
|
||||
code span.sc { color: #4070a0; } /* SpecialChar */
|
||||
code span.ss { color: #bb6688; } /* SpecialString */
|
||||
code span.st { color: #4070a0; } /* String */
|
||||
code span.va { color: #19177c; } /* Variable */
|
||||
code span.vs { color: #4070a0; } /* VerbatimString */
|
||||
code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
|
||||
</style>
|
||||
<link rel="stylesheet" href="../tufte.css" />
|
||||
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js" type="text/javascript"></script>
|
||||
<!--[if lt IE 9]>
|
||||
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div style="display:none">
|
||||
\(
|
||||
\newcommand{\NOT}{\neg}
|
||||
\newcommand{\AND}{\wedge}
|
||||
\newcommand{\OR}{\vee}
|
||||
\newcommand{\XOR}{\oplus}
|
||||
\newcommand{\IMP}{\Rightarrow}
|
||||
\newcommand{\IFF}{\Leftrightarrow}
|
||||
\newcommand{\TRUE}{\text{True}\xspace}
|
||||
\newcommand{\FALSE}{\text{False}\xspace}
|
||||
\newcommand{\IN}{\,{\in}\,}
|
||||
\newcommand{\NOTIN}{\,{\notin}\,}
|
||||
\newcommand{\TO}{\rightarrow}
|
||||
\newcommand{\DIV}{\mid}
|
||||
\newcommand{\NDIV}{\nmid}
|
||||
\newcommand{\MOD}[1]{\pmod{#1}}
|
||||
\newcommand{\MODS}[1]{\ (\text{mod}\ #1)}
|
||||
\newcommand{\N}{\mathbb N}
|
||||
\newcommand{\Z}{\mathbb Z}
|
||||
\newcommand{\Q}{\mathbb Q}
|
||||
\newcommand{\R}{\mathbb R}
|
||||
\newcommand{\C}{\mathbb C}
|
||||
\newcommand{\cA}{\mathcal A}
|
||||
\newcommand{\cB}{\mathcal B}
|
||||
\newcommand{\cC}{\mathcal C}
|
||||
\newcommand{\cD}{\mathcal D}
|
||||
\newcommand{\cE}{\mathcal E}
|
||||
\newcommand{\cF}{\mathcal F}
|
||||
\newcommand{\cG}{\mathcal G}
|
||||
\newcommand{\cH}{\mathcal H}
|
||||
\newcommand{\cI}{\mathcal I}
|
||||
\newcommand{\cJ}{\mathcal J}
|
||||
\newcommand{\cL}{\mathcal L}
|
||||
\newcommand{\cK}{\mathcal K}
|
||||
\newcommand{\cN}{\mathcal N}
|
||||
\newcommand{\cO}{\mathcal O}
|
||||
\newcommand{\cP}{\mathcal P}
|
||||
\newcommand{\cQ}{\mathcal Q}
|
||||
\newcommand{\cS}{\mathcal S}
|
||||
\newcommand{\cT}{\mathcal T}
|
||||
\newcommand{\cV}{\mathcal V}
|
||||
\newcommand{\cW}{\mathcal W}
|
||||
\newcommand{\cZ}{\mathcal Z}
|
||||
\newcommand{\emp}{\emptyset}
|
||||
\newcommand{\bs}{\backslash}
|
||||
\newcommand{\floor}[1]{\left \lfloor #1 \right \rfloor}
|
||||
\newcommand{\ceil}[1]{\left \lceil #1 \right \rceil}
|
||||
\newcommand{\abs}[1]{\left | #1 \right |}
|
||||
\newcommand{\xspace}{}
|
||||
\newcommand{\proofheader}[1]{\underline{\textbf{#1}}}
|
||||
\)
|
||||
</div>
|
||||
<header id="title-block-header">
|
||||
<h1 class="title">8.6 Analyzing Built-In Data Type Operations</h1>
|
||||
</header>
|
||||
<section>
|
||||
<p>So far in our study of running time, we have looked at algorithms that use only primitive numeric data types or loops/comprehensions over collections. In this section, we’re going to study the running time of operations on built-in collection data types (e.g., lists, sets, dictionaries), and the custom data classes that we create. Because a single instance of these compound data types can be very large (e.g., a list of one trillion elements!), the natural question we will ask is, “what operations will take longer when called on very large data structures?” We’ll also study <em>why</em> this is the case for Python lists by studying how they are stored in computer memory. For the other compound data types, however, their implementations are more complex and so we’ll only touch on them in this course.</p>
|
||||
<h2 id="timing-operations">Timing operations</h2>
|
||||
<p>Python provides a module (called <code>timeit</code>) that can tell us how long Python code takes to execute on our machine. Here’s an example showing how to import the module and use it:</p>
|
||||
<div class="sourceCode" id="cb1"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1"></a><span class="op">>>></span> <span class="im">from</span> timeit <span class="im">import</span> timeit</span>
|
||||
<span id="cb1-2"><a href="#cb1-2"></a><span class="op">>>></span> timeit(<span class="st">'5 + 15'</span>, number<span class="op">=</span><span class="dv">1000</span>)</span>
|
||||
<span id="cb1-3"><a href="#cb1-3"></a><span class="fl">1.9799976143985987e-05</span></span></code></pre></div>
|
||||
<p>The call to <code>timeit</code> will perform the operation <code>5 + 15</code> (which we passed in as a string) one thousand times. The function returned the total time elapsed, in seconds, to perform all thousand operations. The return value in the notes is specific to one machine—try the code on your own machine to see how you compare!</p>
|
||||
<p>Next, let’s create two lists with different lengths for comparison: 1,000 and 1,000,000:</p>
|
||||
<div class="sourceCode" id="cb2"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1"></a><span class="op">>>></span> lst_1k <span class="op">=</span> <span class="bu">list</span>(<span class="bu">range</span>(<span class="dv">10</span> <span class="op">**</span> <span class="dv">3</span>))</span>
|
||||
<span id="cb2-2"><a href="#cb2-2"></a><span class="op">>>></span> lst_1m <span class="op">=</span> <span class="bu">list</span>(<span class="bu">range</span>(<span class="dv">10</span> <span class="op">**</span> <span class="dv">6</span>))</span></code></pre></div>
|
||||
<p>We know that there are several operations available to lists. For example, we can search the list using the <code>in</code> operator. Or we could lookup an element at a specific index in the list. Or we could mutate the list by inserting or deleting. Let’s compare the time it takes to access the first element of the list:</p>
|
||||
<div class="sourceCode" id="cb3"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1"></a><span class="op">>>></span> timeit(<span class="st">'lst_1k[0]'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb3-2"><a href="#cb3-2"></a><span class="fl">5.80001506023109e-06</span></span>
|
||||
<span id="cb3-3"><a href="#cb3-3"></a><span class="op">>>></span> timeit(<span class="st">'lst_1m[0]'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb3-4"><a href="#cb3-4"></a><span class="fl">5.599984433501959e-06</span></span></code></pre></div>
|
||||
<p>The length of the list does not seem to impact the time it takes to retrieve an element from this specific index. Let’s compare the time it takes to insert a new element at the front of the list:</p>
|
||||
<div class="sourceCode" id="cb4"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb4-1"><a href="#cb4-1"></a><span class="op">>>></span> timeit(<span class="st">'list.insert(lst_1k, 0, -1)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb4-2"><a href="#cb4-2"></a><span class="fl">0.00014379998901858926</span></span>
|
||||
<span id="cb4-3"><a href="#cb4-3"></a><span class="op">>>></span> timeit(<span class="st">'list.insert(lst_1m, 0, -1)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb4-4"><a href="#cb4-4"></a><span class="fl">0.1726928999996744</span></span></code></pre></div>
|
||||
<p>There is a clear difference in time (by several orders of magnitude) between searching a list with one-thousand elements versus one-million elements.</p>
|
||||
<p>Indeed, every list operation has its own implementation whose running time we can analyze, using the same techniques we studied earlier in this chapter. But in order to fully understand why these implementations work the way they do, we need to dive deeper into how Python lists really work.</p>
|
||||
<h2 id="how-python-lists-are-stored-in-memory">How Python lists are stored in memory</h2>
|
||||
<p>Recall that a Python <code>list</code> object represents an ordered sequence of other objects, which we call its elements. When we studied the object-based memory model in Chapter 5, we drew diagrams like this to represent a <code>list</code>:</p>
|
||||
<div class="fullwidth">
|
||||
<p><img src="images/list_memory_model.png" alt="Memory model diagram of a list" /><br />
|
||||
</p>
|
||||
</div>
|
||||
<p>Our memory-model diagrams are an abstraction. In reality, all data used by a program are stored in blocks of computer memory, which are labeled by numbers called <em>memory addresses</em>, so that the program can keep track of where each piece of data is stored.</p>
|
||||
<p>Here is the key idea for how the Python interpreter stores lists in memory. For every Python <code>list</code> object, the references to its elements are stored in a <em>contiguous</em> block of memory. For example, here is how we could picture the same list as in the previous diagram, now stored in blocks of computer memory:</p>
|
||||
<div class="text-align: center">
|
||||
<p><img src="./images/list_memory_layout.png" style="width:70.0%" alt="Diagram of a list in memory" /><br />
|
||||
</p>
|
||||
</div>
|
||||
<p>As before, our list stores four integers. In memory, the four consecutive blocks 400–403 store references to the actual integer values. Of course, even this diagram is a simplification of what’s actually going on in computer memory, but it illustrates the main point: the references to each list elements are always stored consecutively. This type of list implementation is used by the Python interpreter and many other programming languages, and is called an <strong>array-based list implementation</strong>.</p>
|
||||
<h3 id="fast-list-indexing">Fast list indexing</h3>
|
||||
<p>The primary reason Python uses an array-based list implementation is that it makes list indexing fast. Because the list element references are stored in consecutive memory locations, accessing the <em>i</em>-th element can be done with simple arithmetic: take the memory address where the list starts, and then increase it by <em>i</em> blocks to obtain the the location of the <em>i</em>-th element reference.<label for="sn-0" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-0" class="margin-toggle"/><span class="sidenote"> Think about it like this: suppose you’re walking down a hallway with numbered rooms on just one side and room numbers going up by one. If you see that the first room number is 11, and you’re looking for room 15, you can be confident that it is the fifth room down the hall.</span> More precisely, this means that list indexing is a <em>constant-time</em> operation: its running time does not depend on the size of the list or the index <em>i</em> being accessed. So even with a very long list or a very large index, we expect list indexing to take the same amount of time (and e very fast!).</p>
|
||||
<p>This is true for both evaluating a list indexing expression or assigning to a list index, e.g. <code>lst[1] = 100</code>. In the latter case, the Python interpreter takes constant time to calculate the memory address where the <code>lst[1]</code> reference is stored and modify it to refer to a new object.</p>
|
||||
<h3 id="mutating-contiguous-memory">Mutating contiguous memory</h3>
|
||||
<p>Array-based lists have constant time indexing, but as we’ll see again and again in our study of data types, fast operations almost always come at the cost of slow ones. In order for Python to be able to calculate the address of an arbitrary list index, these references must always be stores in a contiguous block of memory; there can’t be any “gaps”.</p>
|
||||
<p>Maintaining this contiguity has implications for how insertion and deletion in a Python list works. When a list element to be deleted, all items after it have to be moved back one memory block to fill the gap.</p>
|
||||
<div class="text-align: center">
|
||||
<p><img src="images/list_memory_animation_deletion.gif" style="width:70.0%" alt="Animation of deletion in the middle of a list" /><br />
|
||||
</p>
|
||||
</div>
|
||||
<p>Similarly, when a list element is inserted somewhere in the list, all items after it moved forward one block.</p>
|
||||
<div class="text-align: center">
|
||||
<p><img src="images/list_memory_animation_insertion.gif" style="width:70.0%" alt="Animation of insertion in the middle of the list" /><br />
|
||||
</p>
|
||||
</div>
|
||||
<p>In general, suppose we have a list <code>lst</code> of length <span class="math inline">\(n\)</span> and we wish to remove the element at index <span class="math inline">\(i\)</span> in the list, where <span class="math inline">\(0 \leq i < n\)</span>. Then <span class="math inline">\(n - i - 1\)</span> elements must be moved, and the number of “basic operations” this requires is <span class="math inline">\(\Theta(n - i)\)</span>.<label for="sn-1" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-1" class="margin-toggle"/><span class="sidenote"> Here we’re counting moving the contents of one memory block to another as a basic operation.</span> Similarly, if we want to insert an element into a list of length <span class="math inline">\(n\)</span> at index <span class="math inline">\(i\)</span>, <span class="math inline">\(n - i\)</span> elements must be moved, and so the running time of this operation is <span class="math inline">\(\Theta(n - i)\)</span>.</p>
|
||||
<p>At the extremes, this means that inserting/deleting at the front of a Python list (<span class="math inline">\(i = 0\)</span>) takes <span class="math inline">\(\Theta(n)\)</span> time, i.e., proportional to the length of list; on the other hand, inserting/deleting at the back of a Python list (<span class="math inline">\(i = n - 1\)</span>) is a constant-time operation. We can see evidence of this in the following <code>timeit</code> comparisons:</p>
|
||||
<div class="sourceCode" id="cb5"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb5-1"><a href="#cb5-1"></a><span class="op">>>></span> timeit(<span class="st">'list.append(lst_1k, 123)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb5-2"><a href="#cb5-2"></a><span class="fl">1.0400000064691994e-05</span></span>
|
||||
<span id="cb5-3"><a href="#cb5-3"></a><span class="op">>>></span> timeit(<span class="st">'list.append(lst_1m, 123)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb5-4"><a href="#cb5-4"></a><span class="fl">1.3099999932819628e-05</span></span>
|
||||
<span id="cb5-5"><a href="#cb5-5"></a><span class="op">>>></span> timeit(<span class="st">'list.insert(lst_1k, 0, 123)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb5-6"><a href="#cb5-6"></a><span class="fl">4.520000015872938e-05</span></span>
|
||||
<span id="cb5-7"><a href="#cb5-7"></a><span class="op">>>></span> timeit(<span class="st">'list.insert(lst_1m, 0, 123)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb5-8"><a href="#cb5-8"></a><span class="fl">0.011574500000051557</span></span></code></pre></div>
|
||||
<div class="center reference-table">
|
||||
<table>
|
||||
<caption>Summary of list operation asymptotic running times (<span class="math inline">\(n\)</span> is the list size)</caption>
|
||||
<thead>
|
||||
<tr class="header">
|
||||
<th>Operation</th>
|
||||
<th>Running time (<span class="math inline">\(n\)</span> = <code>len(lst)</code>)</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr class="odd">
|
||||
<td>List indexing (<code>lst[i]</code>)</td>
|
||||
<td><span class="math inline">\(\Theta(1)\)</span></td>
|
||||
</tr>
|
||||
<tr class="even">
|
||||
<td>List index assignment (<code>lst[i]</code> = …)</td>
|
||||
<td><span class="math inline">\(\Theta(1)\)</span></td>
|
||||
</tr>
|
||||
<tr class="odd">
|
||||
<td>List insertion at end (<code>list.append(lst, ...)</code>)</td>
|
||||
<td><span class="math inline">\(\Theta(1)\)</span></td>
|
||||
</tr>
|
||||
<tr class="even">
|
||||
<td>List deletion at end (<code>list.pop(lst)</code>)</td>
|
||||
<td><span class="math inline">\(\Theta(1)\)</span></td>
|
||||
</tr>
|
||||
<tr class="odd">
|
||||
<td>List insertion at index (<code>list.insert(lst, i, ...)</code>)</td>
|
||||
<td><span class="math inline">\(\Theta(n - i)\)</span></td>
|
||||
</tr>
|
||||
<tr class="even">
|
||||
<td>List deletion at index (<code>list.pop(lst, i)</code>)</td>
|
||||
<td><span class="math inline">\(\Theta(n - i)\)</span></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<h3 id="when-space-runs-out">When space runs out</h3>
|
||||
<p>Finally, we should point out one subtle assumption we’ve just made in our analysis of list insertion: that there will always be free memory blocks at the end of the list for the list to expand into. In practice, this is almost always true, and so for the purposes of this course we’ll stick with this assumption. But in <em>CSC263/265 (Data Structures and Analysis)</em>, you’ll learn about how programming languages handle array-based list implementations to take into account whether there is “free space” or not, and how these operations still provide the running times we’ve presented in this section.</p>
|
||||
<h2 id="running-time-analysis-with-list-operations">Running-time analysis with list operations</h2>
|
||||
<p>Now that we’ve learned about the running time of basic list operations, let’s see how to apply this knowledge to analysing the running time of algorithms that use these operations. We’ll look at two different examples.</p>
|
||||
<div class="example">
|
||||
<p>Analyse the running time of the following function.</p>
|
||||
<div class="sourceCode" id="cb6"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb6-1"><a href="#cb6-1"></a><span class="kw">def</span> squares(numbers: <span class="bu">list</span>[<span class="bu">int</span>]) <span class="op">-></span> <span class="bu">list</span>[<span class="bu">int</span>]:</span>
|
||||
<span id="cb6-2"><a href="#cb6-2"></a> <span class="co">"""Return a list containing the squares of the given numbers."""</span></span>
|
||||
<span id="cb6-3"><a href="#cb6-3"></a> squares_so_far <span class="op">=</span> []</span>
|
||||
<span id="cb6-4"><a href="#cb6-4"></a></span>
|
||||
<span id="cb6-5"><a href="#cb6-5"></a> <span class="cf">for</span> number <span class="kw">in</span> numbers:</span>
|
||||
<span id="cb6-6"><a href="#cb6-6"></a> <span class="bu">list</span>.append(squares_so_far, number <span class="op">**</span> <span class="dv">2</span>)</span>
|
||||
<span id="cb6-7"><a href="#cb6-7"></a></span>
|
||||
<span id="cb6-8"><a href="#cb6-8"></a> <span class="cf">return</span> squares_so_far</span></code></pre></div>
|
||||
<div class="analysis">
|
||||
<p>Let <span class="math inline">\(n\)</span> be the length of the input list (i.e., <code>numbers</code>).<label for="sn-2" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-2" class="margin-toggle"/><span class="sidenote">Note the similarities between this analysis and our analysis of <code>sum_so_far</code> in <a href="04-basic-algorithm-analysis.html">Section 8.4</a>.</span></p>
|
||||
<p>This function body consists of three statements (with the middle statement, the for loop, itself containing more statements). To analyse the total running time of the function, we need to count each statement separately:</p>
|
||||
<ul>
|
||||
<li>The assignment statement <code>squares_so_far = 0</code> counts as 1 step, as its running time does not depend on the length of <code>numbers</code>.</li>
|
||||
<li>The for loop:
|
||||
<ul>
|
||||
<li>Takes <span class="math inline">\(n\)</span> iterations</li>
|
||||
<li>Inside the loop body, we call <code>list.append(squares_so_far, number ** 2)</code>. Based on our discussion of the previous section, this call to <code>list.append</code> takes <em>constant time</em> (<span class="math inline">\(\Theta(1)\)</span> steps), and so the entire loop body counts as 1 step.</li>
|
||||
<li>This means the for loop takes <span class="math inline">\(n \cdot 1 = n\)</span> steps total.</li>
|
||||
</ul></li>
|
||||
<li>The return statement counts as 1 step: it, too, has running time that does not depend on the length of <code>numbers</code>.</li>
|
||||
</ul>
|
||||
<p>The total running time is the sum of these three parts: <span class="math inline">\(1 + n + 1 = n + 2\)</span>, which is <span class="math inline">\(\Theta(n)\)</span>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p>In our above analysis, we had to take into account the running of calling <code>list.append</code>, but this quantity did not depend on the length of the input list. Our second example will look very similar to the first, but now we use a different <code>list</code> method that results in a dramatic difference in running time:</p>
|
||||
<div class="sourceCode" id="cb7"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb7-1"><a href="#cb7-1"></a><span class="kw">def</span> squares_reversed(numbers: <span class="bu">list</span>[<span class="bu">int</span>]) <span class="op">-></span> <span class="bu">list</span>[<span class="bu">int</span>]:</span>
|
||||
<span id="cb7-2"><a href="#cb7-2"></a> <span class="co">"""Return a list containing the squares of the given numbers, in reverse order."""</span></span>
|
||||
<span id="cb7-3"><a href="#cb7-3"></a> squares_so_far <span class="op">=</span> []</span>
|
||||
<span id="cb7-4"><a href="#cb7-4"></a></span>
|
||||
<span id="cb7-5"><a href="#cb7-5"></a> <span class="cf">for</span> number <span class="kw">in</span> numbers:</span>
|
||||
<span id="cb7-6"><a href="#cb7-6"></a> <span class="co"># Now, insert number ** 2 at the START of squares_so_far</span></span>
|
||||
<span id="cb7-7"><a href="#cb7-7"></a> <span class="bu">list</span>.insert(squares_so_far, <span class="dv">0</span>, number <span class="op">**</span> <span class="dv">2</span>)</span>
|
||||
<span id="cb7-8"><a href="#cb7-8"></a></span>
|
||||
<span id="cb7-9"><a href="#cb7-9"></a> <span class="cf">return</span> squares_so_far</span></code></pre></div>
|
||||
<div class="analysis">
|
||||
<p>Let <span class="math inline">\(n\)</span> be the length of the input list (i.e., <code>numbers</code>).</p>
|
||||
<p>This function body consists of three statements (with the middle statement, the for loop, itself containing more statements). To analyse the total running time of the function, we need to count each statement separately:</p>
|
||||
<ul>
|
||||
<li>The assignment statement <code>squares_so_far = 0</code> counts as 1 step, as its running time does not depend on the length of <code>numbers</code>.</li>
|
||||
<li>The for loop:
|
||||
<ul>
|
||||
<li><p>Takes <span class="math inline">\(n\)</span> iterations</p></li>
|
||||
<li><p>Inside the loop body, we call <code>list.insert(squares_so_far, 0, n ** 2)</code>. As we discussed above, inserting at the front of a Python list causes all of its current elements to be shifted over, taking time proportional to the size of the list. Therefore this call takes <span class="math inline">\(\Theta(k)\)</span> time, where <span class="math inline">\(k\)</span> is the current length of <code>squares_so_far</code>.<label for="sn-3" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-3" class="margin-toggle"/><span class="sidenote"> We can’t use <span class="math inline">\(n\)</span> here, because <span class="math inline">\(n\)</span> already refers to the length of <code>numbers</code>!</span></p>
|
||||
<p>For the purpose of our analysis, we count a function call with <span class="math inline">\(\Theta(k)\)</span> running time as taking <span class="math inline">\(k\)</span> steps, i.e., ignoring the “eventually” and “constant factors” part of the definition of Theta. And so we say that the loop body takes <span class="math inline">\(k\)</span> steps.</p></li>
|
||||
<li><p>In order to calculate the total running time of the loop, we need to add the running times of every iteration. We know that <code>squares_so_far</code> starts as empty, and then increases in length by <code>1</code> at each iteration. So then <span class="math inline">\(k\)</span> (the current length of <code>squares_so_far</code>) takes on the values <span class="math inline">\(0, 1, 2, \dots, n - 1\)</span>, and we can calculate the total running time of the for loop using a summation:</p>
|
||||
<p><span class="math display">\[\sum_{k=0}^{n-1} k = \frac{(n-1)n}{2}\]</span></p></li>
|
||||
</ul></li>
|
||||
<li>The return statement counts as 1 step: it, too, has running time that does not depend on the length of <code>numbers</code>.</li>
|
||||
</ul>
|
||||
<p>The total running time is the sum of these three parts: <span class="math inline">\(1 + \frac{(n-1)n}{2} + 1 = \frac{(n-1)n}{2} + 2\)</span>, which is <span class="math inline">\(\Theta(n^2)\)</span>.</p>
|
||||
</div>
|
||||
<p>To summarize, this single line of code change (from <code>list.append</code> to <code>list.insert</code> at index 0) causes the running time to change dramatically, from <span class="math inline">\(\Theta(n)\)</span> to <span class="math inline">\(\Theta(n^2)\)</span>. When calling functions and performing operations on data types, we must always be conscious of which functions/operations we’re using and their running times. It is easy to skim over a function call because it takes up so little visual space, but that one call might make the difference between running times of <span class="math inline">\(\Theta(n)\)</span>, <span class="math inline">\(\Theta(n^2)\)</span>, or even <span class="math inline">\(\Theta(2^n)\)</span>!<label for="sn-4" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-4" class="margin-toggle"/><span class="sidenote"> Lastly, you might be curious how we could speed up <code>squares_reversed</code>. It turns out that Python has a built-in method <code>list.reverse</code> that mutates a list by reversing it, and this method has a <span class="math inline">\(\Theta(n)\)</span> running time. So we could accumulate the squares by using <code>list.append</code>, and then call <code>list.reverse</code> on the final result.</span></p>
|
||||
<h2 id="sets-and-dictionaries">Sets and dictionaries</h2>
|
||||
<p>It turns out that how Python implements sets and dictionaries is very similar, and so we’ll discuss them together in this section. Both of them are implemented using a more primitive data structure called a <em>hash table</em>, which you’ll also learn about in CSC263/265. The benefit of using hash tables is that they allow <em>constant-time lookup, insertion, and removal</em> of elements (for a set) and key-value pairs (for a dictionary)!<label for="sn-5" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-5" class="margin-toggle"/><span class="sidenote"> This is actually a simplification of how hash tables are implemented. So while we’ll treat all these operations as constant-time in this course, this relies on some technical assumptions which hold in most, but not all, cases.</span></p>
|
||||
<p>But of course, there is a catch. The trade-off of how Python uses hash tables is the elements of a set and the keys of a dictionary cannot be mutable data types, a restriction we discussed earlier in the course. This can be inconvenient, but in general is seen as a small price to pay for the speed of their operations.</p>
|
||||
<p>So if you only care about set operations like “element of”, it is more efficient to use a <code>set</code> than a <code>list</code>:<label for="sn-6" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-6" class="margin-toggle"/><span class="sidenote"> You’ll notice that we haven’t formally discussed the running time of the list <code>in</code> operation in this section. We’ll study it in the next section.</span></p>
|
||||
<div class="sourceCode" id="cb8"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb8-1"><a href="#cb8-1"></a><span class="op">>>></span> lst1M <span class="op">=</span> <span class="bu">list</span>(<span class="bu">range</span>(<span class="dv">10</span> <span class="op">**</span> <span class="dv">6</span>))</span>
|
||||
<span id="cb8-2"><a href="#cb8-2"></a><span class="op">>>></span> set1M <span class="op">=</span> <span class="bu">set</span>(<span class="bu">range</span>(<span class="dv">10</span> <span class="op">**</span> <span class="dv">6</span>))</span>
|
||||
<span id="cb8-3"><a href="#cb8-3"></a><span class="op">>>></span> timeit(<span class="st">'5000000 in lst1M'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb8-4"><a href="#cb8-4"></a><span class="fl">0.16024739999556914</span></span>
|
||||
<span id="cb8-5"><a href="#cb8-5"></a><span class="op">>>></span> timeit(<span class="st">'5000000 in set1M'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb8-6"><a href="#cb8-6"></a><span class="fl">4.6000059228390455e-06</span></span></code></pre></div>
|
||||
<h2 id="data-classes">Data classes</h2>
|
||||
<p>It turns out that data classes (and in fact all Python data types) store their instance attributes using a dictionary that maps attribute names to their corresponding values. This means that data classes benefit from the constant-time dictionary operations that we discussed above.</p>
|
||||
<p>Explicitly, the two operations that we can perform on a data class instance are looking up an attribute value (e.g., <code>david.age</code>) and mutating the instance by assigning to an attribute (e.g., <code>david.age = 99</code>). Both of these operations take constant time, independent of how many instance attributes the data class has or what values are stored for those attributes.</p>
|
||||
<div class="center reference-table">
|
||||
<table>
|
||||
<caption>Summary of set, dictionary, and data class operations</caption>
|
||||
<thead>
|
||||
<tr class="header">
|
||||
<th>Operation</th>
|
||||
<th>Running time</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr class="odd">
|
||||
<td>Set/dict Search (<code>in</code>)</td>
|
||||
<td><span class="math inline">\(\Theta(1)\)</span></td>
|
||||
</tr>
|
||||
<tr class="even">
|
||||
<td><code>set.add</code>/<code>set.remove</code></td>
|
||||
<td><span class="math inline">\(\Theta(1)\)</span></td>
|
||||
</tr>
|
||||
<tr class="odd">
|
||||
<td>Dictionary key lookup (<code>d[k]</code>)</td>
|
||||
<td><span class="math inline">\(\Theta(1)\)</span></td>
|
||||
</tr>
|
||||
<tr class="even">
|
||||
<td>Dictionary key assignment (<code>d[k] = ...</code>)</td>
|
||||
<td><span class="math inline">\(\Theta(1)\)</span></td>
|
||||
</tr>
|
||||
<tr class="odd">
|
||||
<td>Data class attribute access (<code>obj.attr</code>)</td>
|
||||
<td><span class="math inline">\(\Theta(1)\)</span></td>
|
||||
</tr>
|
||||
<tr class="even">
|
||||
<td>Data class attribute assignment (<code>obj.attr = ...</code>)</td>
|
||||
<td><span class="math inline">\(\Theta(1)\)</span></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<h2 id="aggregation-functions">Aggregation functions</h2>
|
||||
<p>Finally, we’ll briefly discuss a few built-in aggregation functions we’ve seen so far in this course.</p>
|
||||
<p><code>sum</code>, <code>max</code>, <code>min</code> have a <em>linear</em> running time (<span class="math inline">\(\Theta(n)\)</span>), proportional to the size of the input collection. This should be fairly intuitive, as each element of the collection must be processed in order to calculate each of these values.</p>
|
||||
<p><code>len</code> is a bit surprising: it has a <em>constant</em> running time (<span class="math inline">\(\Theta(1)\)</span>), independent of the size of the input collection. In order words, the Python interpreter does <em>not</em> need to process each element of a collection when calculating the collection’s size! Instead, each of these collection data types stores a special attribute referring to the size of that collection. And as we discussed for data classes, accessing attributes takes constant time.<label for="sn-7" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-7" class="margin-toggle"/><span class="sidenote"> There is one technical difference between data class attributes and these collection “size” attributes: we can’t access the latter directly in Python code using dot notation, only through calling <code>len</code> on the collection. This is a result of how the Python language implements these built-in collection data types.</span></p>
|
||||
<p><code>any</code> and <code>all</code> are a bit different. Intuitively, they may need to check ever element of their input collection, just like <code>sum</code> or <code>max</code>, but they can also <em>short-circuit</em> (stopping before checking every element), just like the logical <code>or</code> and <code>and</code> operators. This means their running time isn’t a fixed function of the input size, but rather a possible range of values, depending on whether this short-circuiting happens or not. We’ll discuss how to formally analyse the running time of such functions in the next section.</p>
|
||||
</section>
|
||||
<!--
|
||||
### Searching Contiguous Memory
|
||||
|
||||
While contiguous memory helps us find elements at a specific *index*, it does not help with searching elements for a specific *value*.
|
||||
The value being searched for could be at any index (address).
|
||||
The typical algorithm to approach a search is a *linear search*:
|
||||
start at the very first index and check if the value has been found.
|
||||
If not, check the next index and repeat as necessary.
|
||||
We can see that this will take O(n) time, hence why the `in` operator took more time with `lst1M` than `lst1K`.
|
||||
|
||||
-->
|
||||
<footer>
|
||||
<a href="https://www.teach.cs.toronto.edu/~csc110y/fall/notes/">CSC110 Course Notes Home</a>
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,325 @@
|
||||
<!DOCTYPE html>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="generator" content="pandoc" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||||
<title>8.7 Worst-Case Running Time Analysis</title>
|
||||
<style>
|
||||
code{white-space: pre-wrap;}
|
||||
span.smallcaps{font-variant: small-caps;}
|
||||
span.underline{text-decoration: underline;}
|
||||
div.column{display: inline-block; vertical-align: top; width: 50%;}
|
||||
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||||
ul.task-list{list-style: none;}
|
||||
pre > code.sourceCode { white-space: pre; position: relative; }
|
||||
pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
|
||||
pre > code.sourceCode > span:empty { height: 1.2em; }
|
||||
code.sourceCode > span { color: inherit; text-decoration: inherit; }
|
||||
div.sourceCode { margin: 1em 0; }
|
||||
pre.sourceCode { margin: 0; }
|
||||
@media screen {
|
||||
div.sourceCode { overflow: auto; }
|
||||
}
|
||||
@media print {
|
||||
pre > code.sourceCode { white-space: pre-wrap; }
|
||||
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
|
||||
}
|
||||
pre.numberSource code
|
||||
{ counter-reset: source-line 0; }
|
||||
pre.numberSource code > span
|
||||
{ position: relative; left: -4em; counter-increment: source-line; }
|
||||
pre.numberSource code > span > a:first-child::before
|
||||
{ content: counter(source-line);
|
||||
position: relative; left: -1em; text-align: right; vertical-align: baseline;
|
||||
border: none; display: inline-block;
|
||||
-webkit-touch-callout: none; -webkit-user-select: none;
|
||||
-khtml-user-select: none; -moz-user-select: none;
|
||||
-ms-user-select: none; user-select: none;
|
||||
padding: 0 4px; width: 4em;
|
||||
color: #aaaaaa;
|
||||
}
|
||||
pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; }
|
||||
div.sourceCode
|
||||
{ }
|
||||
@media screen {
|
||||
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
|
||||
}
|
||||
code span.al { color: #ff0000; font-weight: bold; } /* Alert */
|
||||
code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
|
||||
code span.at { color: #7d9029; } /* Attribute */
|
||||
code span.bn { color: #40a070; } /* BaseN */
|
||||
code span.bu { } /* BuiltIn */
|
||||
code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
|
||||
code span.ch { color: #4070a0; } /* Char */
|
||||
code span.cn { color: #880000; } /* Constant */
|
||||
code span.co { color: #60a0b0; font-style: italic; } /* Comment */
|
||||
code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
|
||||
code span.do { color: #ba2121; font-style: italic; } /* Documentation */
|
||||
code span.dt { color: #902000; } /* DataType */
|
||||
code span.dv { color: #40a070; } /* DecVal */
|
||||
code span.er { color: #ff0000; font-weight: bold; } /* Error */
|
||||
code span.ex { } /* Extension */
|
||||
code span.fl { color: #40a070; } /* Float */
|
||||
code span.fu { color: #06287e; } /* Function */
|
||||
code span.im { } /* Import */
|
||||
code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
|
||||
code span.kw { color: #007020; font-weight: bold; } /* Keyword */
|
||||
code span.op { color: #666666; } /* Operator */
|
||||
code span.ot { color: #007020; } /* Other */
|
||||
code span.pp { color: #bc7a00; } /* Preprocessor */
|
||||
code span.sc { color: #4070a0; } /* SpecialChar */
|
||||
code span.ss { color: #bb6688; } /* SpecialString */
|
||||
code span.st { color: #4070a0; } /* String */
|
||||
code span.va { color: #19177c; } /* Variable */
|
||||
code span.vs { color: #4070a0; } /* VerbatimString */
|
||||
code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
|
||||
</style>
|
||||
<link rel="stylesheet" href="../tufte.css" />
|
||||
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js" type="text/javascript"></script>
|
||||
<!--[if lt IE 9]>
|
||||
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div style="display:none">
|
||||
\(
|
||||
\newcommand{\NOT}{\neg}
|
||||
\newcommand{\AND}{\wedge}
|
||||
\newcommand{\OR}{\vee}
|
||||
\newcommand{\XOR}{\oplus}
|
||||
\newcommand{\IMP}{\Rightarrow}
|
||||
\newcommand{\IFF}{\Leftrightarrow}
|
||||
\newcommand{\TRUE}{\text{True}\xspace}
|
||||
\newcommand{\FALSE}{\text{False}\xspace}
|
||||
\newcommand{\IN}{\,{\in}\,}
|
||||
\newcommand{\NOTIN}{\,{\notin}\,}
|
||||
\newcommand{\TO}{\rightarrow}
|
||||
\newcommand{\DIV}{\mid}
|
||||
\newcommand{\NDIV}{\nmid}
|
||||
\newcommand{\MOD}[1]{\pmod{#1}}
|
||||
\newcommand{\MODS}[1]{\ (\text{mod}\ #1)}
|
||||
\newcommand{\N}{\mathbb N}
|
||||
\newcommand{\Z}{\mathbb Z}
|
||||
\newcommand{\Q}{\mathbb Q}
|
||||
\newcommand{\R}{\mathbb R}
|
||||
\newcommand{\C}{\mathbb C}
|
||||
\newcommand{\cA}{\mathcal A}
|
||||
\newcommand{\cB}{\mathcal B}
|
||||
\newcommand{\cC}{\mathcal C}
|
||||
\newcommand{\cD}{\mathcal D}
|
||||
\newcommand{\cE}{\mathcal E}
|
||||
\newcommand{\cF}{\mathcal F}
|
||||
\newcommand{\cG}{\mathcal G}
|
||||
\newcommand{\cH}{\mathcal H}
|
||||
\newcommand{\cI}{\mathcal I}
|
||||
\newcommand{\cJ}{\mathcal J}
|
||||
\newcommand{\cL}{\mathcal L}
|
||||
\newcommand{\cK}{\mathcal K}
|
||||
\newcommand{\cN}{\mathcal N}
|
||||
\newcommand{\cO}{\mathcal O}
|
||||
\newcommand{\cP}{\mathcal P}
|
||||
\newcommand{\cQ}{\mathcal Q}
|
||||
\newcommand{\cS}{\mathcal S}
|
||||
\newcommand{\cT}{\mathcal T}
|
||||
\newcommand{\cV}{\mathcal V}
|
||||
\newcommand{\cW}{\mathcal W}
|
||||
\newcommand{\cZ}{\mathcal Z}
|
||||
\newcommand{\emp}{\emptyset}
|
||||
\newcommand{\bs}{\backslash}
|
||||
\newcommand{\floor}[1]{\left \lfloor #1 \right \rfloor}
|
||||
\newcommand{\ceil}[1]{\left \lceil #1 \right \rceil}
|
||||
\newcommand{\abs}[1]{\left | #1 \right |}
|
||||
\newcommand{\xspace}{}
|
||||
\newcommand{\proofheader}[1]{\underline{\textbf{#1}}}
|
||||
\)
|
||||
</div>
|
||||
<header id="title-block-header">
|
||||
<h1 class="title">8.7 Worst-Case Running Time Analysis</h1>
|
||||
</header>
|
||||
<section>
|
||||
<p>In Section 8.3, we saw how to use asymptotic notation to characterize the <em>rate of growth</em> of the number of “basic operations” as a way of analyzing the running time of an algorithm. This approach allows us to ignore details of the computing environment in which the algorithm is run, and machine- and language-dependent implementations of primitive operations, and instead characterize the relationship between the input size and number of basic operations performed.</p>
|
||||
<p>However, this focus on just the input size is a little too restrictive. Even though we can define input size differently for each algorithm we analyze, we tend not to stray too far from the “natural” definitions (e.g., length of list). In practice, though, algorithms often depend on the actual value of the input, not just its size. For example, consider the following function, which searches for an even number in a list of integers.<label for="sn-0" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-0" class="margin-toggle"/><span class="sidenote"> This is very similar to how the <code>in</code> operator is implemented for Python lists.</span></p>
|
||||
<div class="sourceCode" id="cb1"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1"></a><span class="kw">def</span> has_even(numbers: <span class="bu">list</span>[<span class="bu">int</span>]) <span class="op">-></span> <span class="bu">bool</span>:</span>
|
||||
<span id="cb1-2"><a href="#cb1-2"></a> <span class="co">"""Return whether numbers contains an even element."""</span></span>
|
||||
<span id="cb1-3"><a href="#cb1-3"></a> <span class="cf">for</span> number <span class="kw">in</span> numbers:</span>
|
||||
<span id="cb1-4"><a href="#cb1-4"></a> <span class="cf">if</span> number <span class="op">%</span> <span class="dv">2</span> <span class="op">==</span> <span class="dv">0</span>:</span>
|
||||
<span id="cb1-5"><a href="#cb1-5"></a> <span class="cf">return</span> <span class="va">True</span></span>
|
||||
<span id="cb1-6"><a href="#cb1-6"></a> <span class="cf">return</span> <span class="va">False</span></span></code></pre></div>
|
||||
<p>Because this function returns as soon as it finds an even number in the list, its running time is not necessarily proportional to the length of the input list.</p>
|
||||
<p><strong>The running time of a function can vary even when the input size is fixed.</strong> Or using the notation we learned earlier this chapter, the inputs in <span class="math inline">\(\cI_{has\_even, 10}\)</span> do <em>not</em> all have the same runtime. The question “what is <em>the</em> running time of <code>has_even</code> on an input of length <span class="math inline">\(n\)</span>?” does not make sense, as for a given input the runtime depends not just on its length but on which of its elements are even. We illustrate in the following plot, which shows the results of using <code>timeit</code> to measure the running time of <code>has_evens</code> on randomly-chosen lists. While every timing experiment has some inherent uncertainty in the results, the spread of running times cannot be explained by that alone!</p>
|
||||
<p><img src="images/has_evens_plotly.png" style="width:100.0%" alt="Running time plot for has_evens." /><br />
|
||||
</p>
|
||||
<p>Because our asymptotic notation is used to describe the growth rate of <em>functions</em>, we cannot use it to describe the growth of a whole range of values with respect to increasing input sizes. A natural approach to fix this problem is to focus on the <em>maximum</em> of this range, which corresponds to the <em>slowest</em> the algorithm could run for a given input size.</p>
|
||||
<div class="definition" data-terms="worst-case runtime">
|
||||
<p>Let <code>func</code> be a program. We define the function <span class="math inline">\(WC_{func}: \N \to \N\)</span>, called the <strong>worst-case running time function of <code>func</code></strong>, as follows:<label for="sn-1" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-1" class="margin-toggle"/><span class="sidenote">Here, “running time” is measured in exact number of basic operations. We are taking the maximum of a set of numbers, <em>not</em> a set of asymptotic expressions.</span> <span class="math display">\[
|
||||
WC_{func}(n) = \max \big\{ \text{running time of executing $func(x)$} \mid x \in \cI_{func, n} \big\}
|
||||
\]</span></p>
|
||||
</div>
|
||||
<p>Note that <span class="math inline">\(WC_{func}\)</span> is a function, not a (constant) number: it returns the maximum possible running time for an input of size <span class="math inline">\(n\)</span>, for every natural number <span class="math inline">\(n\)</span>. And because it is a function, we can use asymptotic notation to describe it, saying things like “the worst-case running time of this function is <span class="math inline">\(\Theta(n^2)\)</span>.”</p>
|
||||
<p>The goal of a <em>worst-case runtime analysis</em> for <code>func</code> is to find an elementary function <span class="math inline">\(f\)</span> such that <span class="math inline">\(WC_{func} \in \Theta(f)\)</span>.</p>
|
||||
<p>However, it takes a bit more work to obtain tight bounds on a worst-case running time than on the runtime functions of the previous section. It is difficult to compute the <em>exact maximum</em> number of basic operations performed by this algorithm for every input size, which requires that we identify an input for each input size, count its maximum number of basic operations, and then prove that every input of this size takes at most this number of operations. Instead, we will generally take a two-pronged approach: proving matching <em>upper</em> and <em>lower bounds</em> on the worst-case running time of our algorithm.</p>
|
||||
<h3 id="upper-bounds-on-the-worst-case-runtime">Upper bounds on the worst-case runtime</h3>
|
||||
<div class="definition" data-terms="upper bound on worst-case runtime">
|
||||
<p>Let <code>func</code> be a program, and <span class="math inline">\(WC_{func}\)</span> its worst-case runtime function. We say that a function <span class="math inline">\(f: \N \to \R^{\geq 0}\)</span> is an <strong>upper bound on the worst-case runtime</strong> when <span class="math inline">\(WC_{func} \in \cO(f)\)</span>.</p>
|
||||
</div>
|
||||
<p>To get some intuition about what an upper bound on the worst-case running means, suppose we use absolute dominance rather than Big-O. In this case, there’s a very intuitive way to expand the phrase “<span class="math inline">\(WC_{func}\)</span> is absolutely dominated by <span class="math inline">\(f\)</span>”:</p>
|
||||
<p><span class="math display">\[\begin{align*}
|
||||
&\forall n \in \N,~ WC_{func}(n) \leq f(n) \\
|
||||
\Longleftrightarrow \, &\forall n \in \N,~ \max \big\{ \text{running time of executing $func(x)$} \mid x \in \cI_{func, n} \big\} \leq f(n) \\
|
||||
\Longleftrightarrow \, &\forall n \in \N,~ \forall x \in \cI_{func, n},~ \text{running time of executing $func(x)$} \leq f(n)
|
||||
\end{align*}\]</span></p>
|
||||
<p>The last line comes from the fact that if we know the maximum of a set of numbers is less than some value <span class="math inline">\(K\)</span>, then <em>all</em> numbers in that set must be less than <span class="math inline">\(K\)</span>. Thus an upper bound on the worst-case runtime is equivalent to an upper bound on the runtimes of <em>all</em> inputs.</p>
|
||||
<p>Now when we apply the definition of Big-O instead of absolute dominance, we get the following translation of <span class="math inline">\(WC_{func} \in \cO(f)\)</span>:</p>
|
||||
<p><span class="math display">\[
|
||||
\exists c, n_0 \in \R^+,~ \forall n \in \N,~ n \geq n_0 \Rightarrow \big(\forall x \in \cI_{func, n},~ \text{running time of executing $func(x)$} \leq c \cdot f(n) \big)
|
||||
\]</span></p>
|
||||
</div>
|
||||
<p>To approach an analysis of an upper bound on the worst-case, we typically find a function <span class="math inline">\(g\)</span> such that <span class="math inline">\(WC_{func}\)</span> is absolutely dominated by <span class="math inline">\(g\)</span>, and then find a simple function <span class="math inline">\(f\)</span> such that <span class="math inline">\(g \in \cO(f)\)</span>. But how do we find such a <span class="math inline">\(g\)</span>? And what does it mean to upper bound <em>all</em> runtimes of a given input size? We’ll illustrate the technique in our next example.</p>
|
||||
<div id="example:worst_case_upper_bound" class="example">
|
||||
<p>Find an asymptotic upper bound on the worst-case running time of <code>has_even</code>.</p>
|
||||
<div class="discussion">
|
||||
<p>The intuitive translation using absolute dominance is usually enough for an upper bound analysis. In particular, the <span class="math inline">\(\forall n \in \N,~ \forall x \in \cI_{func, n}\)</span> begins with two universal quantifiers, and just knowing this alone should anticipate how we’ll start our proof, using the same techniques of proof we learned earlier!</p>
|
||||
</div>
|
||||
<div class="analysis">
|
||||
<p><em>(Upper bound on worst-case)</em></p>
|
||||
<p>First, let <span class="math inline">\(n \in \N\)</span> and let <code>numbers</code> be an <em>arbitrary</em> list of length <span class="math inline">\(n\)</span>.</p>
|
||||
<p>Now we’ll analyse the running time of <code>has_even</code>, except we <em>can’t</em> assume anything about the values inside <code>numbers</code>, because it’s an aribtrary list. But we can still find an <em>upper bound</em> on the running time:</p>
|
||||
<ul>
|
||||
<li><p>The loop (<code>for number in numbers</code>) iterates <em>at most</em> <span class="math inline">\(n\)</span> times. Each loop iteration counts as a single step (because it is constant time), so the loop takes <em>at most</em> <span class="math inline">\(n \cdot 1 = n\)</span> steps in total.</p></li>
|
||||
<li><p>The <code>return False</code> statement (if it is executed) counts as <span class="math inline">\(1\)</span> basic operation.</p></li>
|
||||
</ul>
|
||||
<p>Therefore the running time is <em>at most <span class="math inline">\(n + 1\)</span></em>, and <span class="math inline">\(n + 1 \in \cO(n)\)</span>. So we can conclude that the worst-case running time of <code>has_even</code> is <span class="math inline">\(\cO(n)\)</span>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p>Note that we did <em>not</em> prove that <code>has_even(numbers)</code> takes exactly <span class="math inline">\(n + 1\)</span> basic operations for an arbitrary input <code>numbers</code> (this is false); we only proved an <em>upper bound</em> on the number of operations. And in fact, we don’t even care that much about the exact number: what we ultimately care about is the asymptotic growth rate, which is linear for <span class="math inline">\(n + 1\)</span>. This allowed us to conclude that the worst-case running time of <code>has_even</code> is <span class="math inline">\(\cO(n)\)</span>.</p>
|
||||
<p>But because we calculated an upper bound rather than an exact number of steps, we can only conclude a Big-O, not Theta bound: we don’t yet know that this upper bound is tight.<label for="sn-2" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-2" class="margin-toggle"/><span class="sidenote">If this is surprising, note that we could have done the above proof but replaced <span class="math inline">\(n+1\)</span> by <span class="math inline">\(5000n + 110\)</span> and it would still have been mathematically valid.</span></p>
|
||||
<h3 id="lower-bounds-on-the-worst-case-runtime">Lower bounds on the worst-case runtime</h3>
|
||||
<p>So how do we prove our upper bound is tight? Since we’ve just shown that <span class="math inline">\(WC_{has\_even}(n) \in \cO(n)\)</span>, we need to prove the corresponding lower bound <span class="math inline">\(WC_{has\_even}(n) \in \Omega(n)\)</span>. But what does it mean to prove a lower bound on the maximum of a set of numbers? Suppose we have a set of numbers <span class="math inline">\(S\)</span>, and say that “the maximum of <span class="math inline">\(S\)</span> is at least <span class="math inline">\(50\)</span>.” This doesn’t tell us what the maximum of <span class="math inline">\(S\)</span> actually is, but it does give us one piece of information: there has to be a number in <span class="math inline">\(S\)</span> which is at least <span class="math inline">\(50\)</span>.</p>
|
||||
<p>The key insight is that the converse is also true—if I tell you that <span class="math inline">\(S\)</span> contains the number <span class="math inline">\(50\)</span>, then you can conclude that the maximum of <span class="math inline">\(S\)</span> is at least <span class="math inline">\(50\)</span>. <span class="math display">\[\max(S) \geq 50 \IFF (\exists x \in S,~ x \geq 50)\]</span> Using this idea, we’ll give a formal definition for a lower bound on the worst-case runtime of an algorithm.</p>
|
||||
<div class="definition" data-terms="lower bound on worst-case runtime">
|
||||
<p>Let <code>func</code> be a program, and <span class="math inline">\(WC_{func}\)</span> is worst-case runtime function. We say that a function <span class="math inline">\(f: \N \to \R^{\geq 0}\)</span> is a <strong>lower bound on the worst-case runtime</strong> when <span class="math inline">\(WC_{func} \in \Omega(f)\)</span>.</p>
|
||||
<p>In an analogous fashion to the upper bound, we unpack this definition first by using absolute dominance: <span class="math display">\[\begin{align*}
|
||||
& \forall n \in \N,~ WC_{func}(n) \geq f(n) \\
|
||||
\Longleftrightarrow \, &\forall n \in \N,~ \max \big\{ \text{running time of executing $func(x)$} \mid x \in \cI_{func, n} \big\} \geq f(n) \\
|
||||
\Longleftrightarrow \, &\forall n \in \N,~ \exists x \in \cI_{func, n},~ \text{running time of executing $func(x)$} \geq f(n)
|
||||
\end{align*}\]</span></p>
|
||||
<p>And then using Omega:</p>
|
||||
<p><span class="math display">\[
|
||||
\exists c, n_0 \in \R^+,~ \forall n \in \N,~ n \geq n_0 \Rightarrow
|
||||
\big(\exists x \in \cI_{func, n},~ \text{running time of executing $func(x)$} \geq c \cdot f(n) \big)
|
||||
\]</span></p>
|
||||
</div>
|
||||
<p>Remarkably, the crucial difference between this definition and the one for upper bounds is a change of quantifier: now the input <span class="math inline">\(x\)</span> is existentially quantified, meaning we get to pick it. Or really, our goal is to find an <strong>input family</strong>—a <em>set</em> of inputs, one per input size <span class="math inline">\(n\)</span>—whose runtime is asymptotically larger than our target lower bound.</p>
|
||||
<p>For example, in <code>has_even</code> we want to prove that the worst-case running time is <span class="math inline">\(\Omega(n)\)</span> to match the <span class="math inline">\(\cO(n)\)</span> upper bound, and so we want to find and input family where the number of steps taken is <span class="math inline">\(\Omega(n)\)</span>. Let’s do that now.</p>
|
||||
<div id="example:worst_case_lower_bound" class="example">
|
||||
<p>Find an asymptotic lower bound on the worst-case running time of <code>has_even</code>.</p>
|
||||
<div class="discussion">
|
||||
<p>Again, we’ll just remind you of the quantifiers from the intuitive “absolute dominance” version of the lower bound definition: <span class="math inline">\(\forall n \in \N,~ \exists x \in \cI_{n}\)</span>. This will inform how we start our proof.</p>
|
||||
</div>
|
||||
<div class="analysis">
|
||||
<p><em>(Lower bound on worst-case)</em></p>
|
||||
<p>Let <span class="math inline">\(n \in \N\)</span>. Let <code>numbers</code> be the list of length <span class="math inline">\(n\)</span> consisting of all <span class="math inline">\(1\)</span>’s. Now we’ll analyse the (exact) running time of <code>has_even</code> on this input.</p>
|
||||
<p>In this case, the <code>if</code> condition in the loop is always false, so the loop never stops early. Therefore it iterates exactly <span class="math inline">\(n\)</span> times (once per item in the list), with each iteration taking one step.</p>
|
||||
<p>Finally, the <code>return False</code> statement executes, which is one step. So the total number of steps for this input is <span class="math inline">\(n + 1\)</span>, which is <span class="math inline">\(\Omega(n)\)</span>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<h3 id="putting-it-all-together">Putting it all together</h3>
|
||||
<p>Finally, we can combine our upper and lower bounds on <span class="math inline">\(WC_{has\_even}\)</span> to obtain a tight asymptotic bound.</p>
|
||||
<div class="example">
|
||||
<p>Find a <em>tight</em> bound on the worst-case running time of <code>has_even</code>.</p>
|
||||
<div class="analysis">
|
||||
<p>Since we’ve proved that <span class="math inline">\(WC_{has\_even}\)</span> is <span class="math inline">\(\cO(n)\)</span> and <span class="math inline">\(\Omega(n)\)</span>, it is <span class="math inline">\(\Theta(n)\)</span>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p>To summarize, to obtain a tight bound on the worst-case running time of a function, we need to do two things:</p>
|
||||
<ul>
|
||||
<li><p>Use the properties of the code to obtain an <em>asymptotic upper bound</em> on the worst-case running time. We would say something like <span class="math inline">\(WC_{func} \in \cO(f)\)</span>.</p></li>
|
||||
<li><p>Find a family of inputs whose running time is <span class="math inline">\(\Omega(f)\)</span>.<label for="sn-3" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-3" class="margin-toggle"/><span class="sidenote"> Almost always we find an input family whose running time is <span class="math inline">\(\Theta(f)\)</span>, but strictly speaking only <span class="math inline">\(\Omega(f)\)</span> is required.</span> This will prove that <span class="math inline">\(WC_{func} \in \Omega(f)\)</span>.</p></li>
|
||||
<li><p>After showing that <span class="math inline">\(WC_{func} \in \cO(f)\)</span> and <span class="math inline">\(WC_{func} \in \Omega(f)\)</span>, we can conclude that <span class="math inline">\(WC_f \in \Theta(f)\)</span>.</p></li>
|
||||
</ul>
|
||||
<h3 id="a-note-about-best-case-runtime">A note about best-case runtime</h3>
|
||||
<p>In this section, we focused on worst-case runtime, the result of taking the <em>maximum</em> runtime for every input size. It is also possible to define a best-case runtime function by taking the minimum possible runtimes, and obtain tight bounds on the best case through an analysis that is completely analogous to the one we just performed. In practice, however, the best-case runtime of an algorithm is usually not as useful to know—we care far more about knowing just how <em>slow</em> an algorithm is than how fast it can be.</p>
|
||||
<h2 id="early-returning-in-python-built-ins">Early returning in Python built-ins</h2>
|
||||
<p>We’ve encountered a few different Python functions and methods whose running time depends on more than just the size of their inputs. We alluded to one at the start of this chapter: the list search operation using the keyword <code>in</code>:</p>
|
||||
<div class="sourceCode" id="cb2"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1"></a><span class="op">>>></span> lst <span class="op">=</span> <span class="bu">list</span>(<span class="bu">range</span>(<span class="dv">0</span>, <span class="dv">1000000</span>))</span>
|
||||
<span id="cb2-2"><a href="#cb2-2"></a><span class="op">>>></span> timeit.timeit(<span class="st">'0 in lst'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb2-3"><a href="#cb2-3"></a><span class="fl">8.299997716676444e-06</span></span>
|
||||
<span id="cb2-4"><a href="#cb2-4"></a><span class="op">>>></span> timeit.timeit(<span class="st">'-1 in lst'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb2-5"><a href="#cb2-5"></a><span class="fl">0.17990550000104122</span></span></code></pre></div>
|
||||
<p>In the first <code>timeit</code> expression, <code>0</code> appears as the first element of <code>lst</code>, and so is found immediately when the search occurs. In the second, <code>-1</code> does not appear in <code>lst</code> at all, and so all one-million elements of <code>lst</code> must be checked, resulting in a running-time that is proportional to the length of the list. <em>The worst-case running time of the <code>in</code> operation for lists is <span class="math inline">\(\Theta(n)\)</span>, where <span class="math inline">\(n\)</span> is the length of the list.</em></p>
|
||||
<p>We have also seen two more functions that are implemented using an early return: <code>any</code> and <code>all</code>. Because <code>any</code> searches for a single <code>True</code> in a collection, it stops the first time it finds one. Similarly, because <code>all</code> requires that all elements of a collection be <code>True</code>, it stops the first time it finds a <code>False</code> value.</p>
|
||||
<div class="sourceCode" id="cb3"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1"></a><span class="op">>>></span> all_trues <span class="op">=</span> [<span class="va">True</span>] <span class="op">*</span> <span class="dv">1000000</span></span>
|
||||
<span id="cb3-2"><a href="#cb3-2"></a><span class="op">>>></span> all_falses <span class="op">=</span> [<span class="va">False</span>] <span class="op">*</span> <span class="dv">1000000</span></span>
|
||||
<span id="cb3-3"><a href="#cb3-3"></a><span class="op">>>></span> timeit.timeit(<span class="st">'any(all_trues)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb3-4"><a href="#cb3-4"></a><span class="fl">8.600000001024455e-06</span></span>
|
||||
<span id="cb3-5"><a href="#cb3-5"></a><span class="op">>>></span> timeit.timeit(<span class="st">'any(all_falses)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb3-6"><a href="#cb3-6"></a><span class="fl">0.10643419999905745</span></span>
|
||||
<span id="cb3-7"><a href="#cb3-7"></a><span class="op">>>></span> timeit.timeit(<span class="st">'all(all_trues)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb3-8"><a href="#cb3-8"></a><span class="fl">0.10217570000168053</span></span>
|
||||
<span id="cb3-9"><a href="#cb3-9"></a><span class="op">>>></span> timeit.timeit(<span class="st">'all(all_falses)'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb3-10"><a href="#cb3-10"></a><span class="fl">6.300000677583739e-06</span></span></code></pre></div>
|
||||
<p>So in the above example:</p>
|
||||
<ul>
|
||||
<li><code>any(all_trues)</code> returns <code>True</code> immediately after checking the first list element.</li>
|
||||
<li><code>any(all_falses)</code> returns <code>False</code> only after checking all one-million list elements.</li>
|
||||
<li><code>all(all_trues)</code> returns <code>True</code> only after checking all one-million list elements.</li>
|
||||
<li><code>all(all_falses)</code> returns <code>False</code> immediately after checking the first list element.</li>
|
||||
</ul>
|
||||
<p>So <code>any</code> and <code>all</code> have a worst-case running time of <span class="math inline">\(\Theta(n)\)</span>, where <span class="math inline">\(n\)</span> is the size of the input collection. But in practice they can be much faster if they encounter the “right” boolean value early on!</p>
|
||||
<h3 id="any-all-and-comprehensions"><code>any</code>, <code>all</code>, and comprehensions</h3>
|
||||
<p>There is one subtlety that often catches students by surprise when they attempt to call <code>any</code>/<code>all</code> on a comprehension and expect a quick result. Let’s see a simple example:</p>
|
||||
<div class="sourceCode" id="cb4"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb4-1"><a href="#cb4-1"></a><span class="op">>>></span> timeit.timeit(<span class="st">'any([x == 0 for x in range(0, 1000000)])'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb4-2"><a href="#cb4-2"></a><span class="fl">0.7032962000012049</span></span></code></pre></div>
|
||||
<p>That’s a lot slower than we would expect, given that the first element checked is <code>x = 0</code>! The result is similar if we try to use a set comprehension instead of a list comprehension:</p>
|
||||
<div class="sourceCode" id="cb5"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb5-1"><a href="#cb5-1"></a><span class="op">>>></span> timeit.timeit(<span class="st">'any({x == 0 for x in range(0, 1000000)})'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb5-2"><a href="#cb5-2"></a><span class="fl">0.6538308000017423</span></span></code></pre></div>
|
||||
<p>The subtlety here is that in both cases, <em>the full comprehension is evaluated before <code>any</code> is called</em>. As we discussed in <a href="05-more-runtime.html">8.5 Analyzing Comprehensions and While Loops</a>, the running time of evaluating a comprehension is proportional to the size of the source collection of the comprehension—in our example, that’s <code>range(0, 1000000)</code>, which contains one-million numbers.</p>
|
||||
<p>But all is not lost! In practice, Python programmers <em>do</em> use <code>any</code>/<code>all</code> with comprehensions, but they do so by writing the comprehension expression in the function call without any surrounding square brackets or curly braces:</p>
|
||||
<div class="sourceCode" id="cb6"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb6-1"><a href="#cb6-1"></a><span class="op">>>></span> <span class="bu">any</span>(x <span class="op">==</span> <span class="dv">0</span> <span class="cf">for</span> x <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, <span class="dv">1000000</span>))</span>
|
||||
<span id="cb6-2"><a href="#cb6-2"></a><span class="va">True</span></span></code></pre></div>
|
||||
<p>This is called a <strong>generator comprehension</strong>, and is used to produce a special Python collection data type called a <strong>generator</strong>. We won’t use generators or generator comprehensions very much at all in this course, but what we want you to know about them here is that unlike set/list comprehensions, generator comprehensions do not evaluate their elements all at once, but instead only when they are needed by the function being called. This means that our above <code>any</code> call achieves the fast running time we initially expected:</p>
|
||||
<div class="sourceCode" id="cb7"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb7-1"><a href="#cb7-1"></a><span class="op">>>></span> timeit.timeit(<span class="st">'any(x == 0 for x in range(0, 1000000))'</span>, number<span class="op">=</span><span class="dv">10</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb7-2"><a href="#cb7-2"></a><span class="fl">4.050000279676169e-05</span></span></code></pre></div>
|
||||
<p>Now, only the <code>x = 0</code> value from the generator comprehension gets evaluated; none of the other possible values (<code>x = 1, 2, ..., 999999</code>) are ever checked by the <code>any</code> call!</p>
|
||||
<h2 id="dont-assume-bounds-are-tight">Don’t assume bounds are tight!</h2>
|
||||
<p>It is likely unsatisfying to hear that upper and lower bounds really are distinct things that must be computed separately. Our intuition here pulls us towards the bounds being “obviously” the same, but this is really a side effect of the examples we have studied so far in this course being rather straightforward. But this won’t always be the case: the study of more complex algorithms and data structures exhibits quite a few situations where obtaining an upper bound on the running time involves a completely different analysis than the lower bound.</p>
|
||||
<p>Let’s look at one such example that deals with manipulating strings.</p>
|
||||
<div class="example">
|
||||
<p>We say that a string is a <strong>palindrome</strong> when it can be read the same forwards and backwards; example of palindromes are “abba”, “racecar”, and “z”.<label for="sn-4" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-4" class="margin-toggle"/><span class="sidenote">Every string of length 1 is a palindrome.</span> We say that a string <span class="math inline">\(s_1\)</span> is a <strong>prefix</strong> of another string <span class="math inline">\(s_2\)</span> when <span class="math inline">\(s_1\)</span> is a substring of <span class="math inline">\(s_2\)</span> that starts at index 0 of <span class="math inline">\(s_2\)</span>. For example, the string “abc” is a prefix of “abcdef”.</p>
|
||||
<p>The algorithm below takes a non-empty string as input, and returns the length of the longest prefix of that string that is a palindrome. For example, the string “attack” has two non-empty prefixes that are palindromes, “a” and “atta”, and so our algorithm will return 4.</p>
|
||||
<div class="sourceCode" id="cb8"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb8-1"><a href="#cb8-1"></a><span class="kw">def</span> palindrome_prefix(s: <span class="bu">str</span>) <span class="op">-></span> <span class="bu">int</span>:</span>
|
||||
<span id="cb8-2"><a href="#cb8-2"></a> n <span class="op">=</span> <span class="bu">len</span>(s)</span>
|
||||
<span id="cb8-3"><a href="#cb8-3"></a> <span class="cf">for</span> prefix_length <span class="kw">in</span> <span class="bu">range</span>(n, <span class="dv">0</span>, <span class="op">-</span><span class="dv">1</span>): <span class="co"># goes from n down to 1</span></span>
|
||||
<span id="cb8-4"><a href="#cb8-4"></a> <span class="co"># Check whether s[0:prefix_length] is a palindrome</span></span>
|
||||
<span id="cb8-5"><a href="#cb8-5"></a> is_palindrome <span class="op">=</span> <span class="bu">all</span>(s[i] <span class="op">==</span> s[prefix_length <span class="op">-</span> <span class="dv">1</span> <span class="op">-</span> i]</span>
|
||||
<span id="cb8-6"><a href="#cb8-6"></a> <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, prefix_length))</span>
|
||||
<span id="cb8-7"><a href="#cb8-7"></a></span>
|
||||
<span id="cb8-8"><a href="#cb8-8"></a> <span class="co"># If a palindrome prefix is found, return the current length.</span></span>
|
||||
<span id="cb8-9"><a href="#cb8-9"></a> <span class="cf">if</span> is_palindrome:</span>
|
||||
<span id="cb8-10"><a href="#cb8-10"></a> <span class="cf">return</span> prefix_length</span></code></pre></div>
|
||||
<p>There are a few interesting details to note about this algorithm:</p>
|
||||
<ul>
|
||||
<li>The for loop iterable is <code>range(n, 0, -1)</code>—the third argument <code>-1</code> causes the loop variable to <em>start</em> at <code>n</code> and decrease by 1 at each iteration. In other words, this loop is checking the possible prefixes starting with the longest prefix (length <code>n</code>) and working its way to the shortest prefix (length 1).</li>
|
||||
<li>The call to <code>all</code> checks pairs of characters starting at either end of the current prefix. It uses a <em>generator comprehension</em> (like we discussed above) so that it can stop early as soon as it encounters a mismatch (i.e., when <code>s[i] != s[prefix_length - 1 - i]</code>).</li>
|
||||
<li>Even though the only return statement is inside the <code>for</code> loop, this algorithm is guaranteed to find a palindrome prefix, since the first letter of <code>s</code> by itself is a palindrome.</li>
|
||||
</ul>
|
||||
<p>The code presented here is structurally simple. Indeed, it is not too hard to show that the worst-case runtime of this function is <span class="math inline">\(\cO(n^2)\)</span>, where <span class="math inline">\(n\)</span> is the length of the input string. What is harder, however, is showing that the worst-case runtime is <span class="math inline">\(\Omega(n^2)\)</span>. To do so, we must find an input family whose runtime is <span class="math inline">\(\Omega(n^2)\)</span>. There are two points in the code that can lead to fewer than the maximum loop iterations occurring, and we want to find an input family that avoids both of these.</p>
|
||||
<p>The difficulty is that these two points are caused by different types of inputs! The call to <code>all</code> can stop as soon as the algorithm detects that a prefix is <em>not</em> a palindrome, while the <code>return</code> statement occurs when the algorithm has determined that a prefix <em>is</em> a palindrome! To make this tension more explicit, let’s consider two extreme input families that seem plausible at first glance, but which do not have a runtime that is <span class="math inline">\(\Omega(n^2)\)</span>.</p>
|
||||
<ul>
|
||||
<li>The entire string <span class="math inline">\(s\)</span> is a palindrome of length <span class="math inline">\(n\)</span>. In this case, in the first iteration of the loop, the entire string is checked. The <code>all</code> call checks all pairs of characters, but unfortunately this means that <code>is_palindrome = True</code>, and the loop returns during its very first iteration. Since the <code>all</code> call takes <span class="math inline">\(n\)</span> steps, this input family takes <span class="math inline">\(\Theta(n)\)</span> time to run.</li>
|
||||
<li>The entire string <span class="math inline">\(s\)</span> consists of <span class="math inline">\(n\)</span> different letters. In this case, the only palindrome prefix is just the first letter of <span class="math inline">\(s\)</span> itself. This means that the loop will run for all <span class="math inline">\(n\)</span> iterations, only returning in its last iteration (when <code>prefix_length == 1</code>). However, the <code>all</code> call will always stop after just one step, since it starts by comparing the first letter of <span class="math inline">\(s\)</span> with another letter, which is guaranteed to be different by our choice of input family. This again leads to a <span class="math inline">\(\Theta(n)\)</span> running time.</li>
|
||||
</ul>
|
||||
<p>The key idea is that we want to choose an input family that <em>doesn’t</em> contain a long palindrome (so the loop runs for many iterations), but whose prefixes are close to being palindromes like palindromes (so the <code>all</code> call checks many pairs of letters). Let <span class="math inline">\(n \in \Z^+\)</span>. We define the input <span class="math inline">\(s_n\)</span> as follows:</p>
|
||||
<ul>
|
||||
<li><span class="math inline">\(s_n[\ceil{n/2}] = b\)</span></li>
|
||||
<li>Every other character in <span class="math inline">\(s_n\)</span> is equal to <span class="math inline">\(a\)</span>.</li>
|
||||
</ul>
|
||||
<p>For example, <span class="math inline">\(s_4 = aaba\)</span> and <span class="math inline">\(s_{11} = aaaaaabaaa\)</span>. Note that <span class="math inline">\(s_n\)</span> is very close to being a palindrome: if that single character <span class="math inline">\(b\)</span> were changed to an <span class="math inline">\(a\)</span>, then <span class="math inline">\(s_n\)</span> would be the all-<span class="math inline">\(a\)</span>’s string, which is certainly a palindrome. But by making the centre character a <span class="math inline">\(b\)</span>, we not only ensure that the longest palindrome of <span class="math inline">\(s_n\)</span> has length roughly <span class="math inline">\(n/2\)</span> (so the loop iterates roughly <span class="math inline">\(n/2\)</span> times), but also that the “outer” characters of each prefix of <span class="math inline">\(s_n\)</span> containing more than <span class="math inline">\(n/2\)</span> characters are all the same (so the <code>all</code> call checks many pairs to find the mismatch between <span class="math inline">\(a\)</span> and <span class="math inline">\(b\)</span>). It turns out that this input family does indeed have an <span class="math inline">\(\Omega(n^2)\)</span> runtime! We’ll leave the details as an exercise.</p>
|
||||
</div>
|
||||
</section>
|
||||
<footer>
|
||||
<a href="https://www.teach.cs.toronto.edu/~csc110y/fall/notes/">CSC110 Course Notes Home</a>
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,212 @@
|
||||
<!DOCTYPE html>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="generator" content="pandoc" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||||
<title>8.8 Testing Functions IV: Efficiency</title>
|
||||
<style>
|
||||
code{white-space: pre-wrap;}
|
||||
span.smallcaps{font-variant: small-caps;}
|
||||
span.underline{text-decoration: underline;}
|
||||
div.column{display: inline-block; vertical-align: top; width: 50%;}
|
||||
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||||
ul.task-list{list-style: none;}
|
||||
pre > code.sourceCode { white-space: pre; position: relative; }
|
||||
pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
|
||||
pre > code.sourceCode > span:empty { height: 1.2em; }
|
||||
code.sourceCode > span { color: inherit; text-decoration: inherit; }
|
||||
div.sourceCode { margin: 1em 0; }
|
||||
pre.sourceCode { margin: 0; }
|
||||
@media screen {
|
||||
div.sourceCode { overflow: auto; }
|
||||
}
|
||||
@media print {
|
||||
pre > code.sourceCode { white-space: pre-wrap; }
|
||||
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
|
||||
}
|
||||
pre.numberSource code
|
||||
{ counter-reset: source-line 0; }
|
||||
pre.numberSource code > span
|
||||
{ position: relative; left: -4em; counter-increment: source-line; }
|
||||
pre.numberSource code > span > a:first-child::before
|
||||
{ content: counter(source-line);
|
||||
position: relative; left: -1em; text-align: right; vertical-align: baseline;
|
||||
border: none; display: inline-block;
|
||||
-webkit-touch-callout: none; -webkit-user-select: none;
|
||||
-khtml-user-select: none; -moz-user-select: none;
|
||||
-ms-user-select: none; user-select: none;
|
||||
padding: 0 4px; width: 4em;
|
||||
color: #aaaaaa;
|
||||
}
|
||||
pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; }
|
||||
div.sourceCode
|
||||
{ }
|
||||
@media screen {
|
||||
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
|
||||
}
|
||||
code span.al { color: #ff0000; font-weight: bold; } /* Alert */
|
||||
code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
|
||||
code span.at { color: #7d9029; } /* Attribute */
|
||||
code span.bn { color: #40a070; } /* BaseN */
|
||||
code span.bu { } /* BuiltIn */
|
||||
code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
|
||||
code span.ch { color: #4070a0; } /* Char */
|
||||
code span.cn { color: #880000; } /* Constant */
|
||||
code span.co { color: #60a0b0; font-style: italic; } /* Comment */
|
||||
code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
|
||||
code span.do { color: #ba2121; font-style: italic; } /* Documentation */
|
||||
code span.dt { color: #902000; } /* DataType */
|
||||
code span.dv { color: #40a070; } /* DecVal */
|
||||
code span.er { color: #ff0000; font-weight: bold; } /* Error */
|
||||
code span.ex { } /* Extension */
|
||||
code span.fl { color: #40a070; } /* Float */
|
||||
code span.fu { color: #06287e; } /* Function */
|
||||
code span.im { } /* Import */
|
||||
code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
|
||||
code span.kw { color: #007020; font-weight: bold; } /* Keyword */
|
||||
code span.op { color: #666666; } /* Operator */
|
||||
code span.ot { color: #007020; } /* Other */
|
||||
code span.pp { color: #bc7a00; } /* Preprocessor */
|
||||
code span.sc { color: #4070a0; } /* SpecialChar */
|
||||
code span.ss { color: #bb6688; } /* SpecialString */
|
||||
code span.st { color: #4070a0; } /* String */
|
||||
code span.va { color: #19177c; } /* Variable */
|
||||
code span.vs { color: #4070a0; } /* VerbatimString */
|
||||
code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
|
||||
</style>
|
||||
<link rel="stylesheet" href="../tufte.css" />
|
||||
<!--[if lt IE 9]>
|
||||
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div style="display:none">
|
||||
\(
|
||||
\newcommand{\NOT}{\neg}
|
||||
\newcommand{\AND}{\wedge}
|
||||
\newcommand{\OR}{\vee}
|
||||
\newcommand{\XOR}{\oplus}
|
||||
\newcommand{\IMP}{\Rightarrow}
|
||||
\newcommand{\IFF}{\Leftrightarrow}
|
||||
\newcommand{\TRUE}{\text{True}\xspace}
|
||||
\newcommand{\FALSE}{\text{False}\xspace}
|
||||
\newcommand{\IN}{\,{\in}\,}
|
||||
\newcommand{\NOTIN}{\,{\notin}\,}
|
||||
\newcommand{\TO}{\rightarrow}
|
||||
\newcommand{\DIV}{\mid}
|
||||
\newcommand{\NDIV}{\nmid}
|
||||
\newcommand{\MOD}[1]{\pmod{#1}}
|
||||
\newcommand{\MODS}[1]{\ (\text{mod}\ #1)}
|
||||
\newcommand{\N}{\mathbb N}
|
||||
\newcommand{\Z}{\mathbb Z}
|
||||
\newcommand{\Q}{\mathbb Q}
|
||||
\newcommand{\R}{\mathbb R}
|
||||
\newcommand{\C}{\mathbb C}
|
||||
\newcommand{\cA}{\mathcal A}
|
||||
\newcommand{\cB}{\mathcal B}
|
||||
\newcommand{\cC}{\mathcal C}
|
||||
\newcommand{\cD}{\mathcal D}
|
||||
\newcommand{\cE}{\mathcal E}
|
||||
\newcommand{\cF}{\mathcal F}
|
||||
\newcommand{\cG}{\mathcal G}
|
||||
\newcommand{\cH}{\mathcal H}
|
||||
\newcommand{\cI}{\mathcal I}
|
||||
\newcommand{\cJ}{\mathcal J}
|
||||
\newcommand{\cL}{\mathcal L}
|
||||
\newcommand{\cK}{\mathcal K}
|
||||
\newcommand{\cN}{\mathcal N}
|
||||
\newcommand{\cO}{\mathcal O}
|
||||
\newcommand{\cP}{\mathcal P}
|
||||
\newcommand{\cQ}{\mathcal Q}
|
||||
\newcommand{\cS}{\mathcal S}
|
||||
\newcommand{\cT}{\mathcal T}
|
||||
\newcommand{\cV}{\mathcal V}
|
||||
\newcommand{\cW}{\mathcal W}
|
||||
\newcommand{\cZ}{\mathcal Z}
|
||||
\newcommand{\emp}{\emptyset}
|
||||
\newcommand{\bs}{\backslash}
|
||||
\newcommand{\floor}[1]{\left \lfloor #1 \right \rfloor}
|
||||
\newcommand{\ceil}[1]{\left \lceil #1 \right \rceil}
|
||||
\newcommand{\abs}[1]{\left | #1 \right |}
|
||||
\newcommand{\xspace}{}
|
||||
\newcommand{\proofheader}[1]{\underline{\textbf{#1}}}
|
||||
\)
|
||||
</div>
|
||||
<header id="title-block-header">
|
||||
<h1 class="title">8.8 Testing Functions IV: Efficiency</h1>
|
||||
</header>
|
||||
<section>
|
||||
<p>We hope that the number of sections of these notes dedicated to testing demonstrates its importance in the process of software development. What is perhaps surprising is that testing is not limited to correctness. In fact, strict efficiency constraints are the norm in several domains. For example, a Playstation controller must send wireless signals at the touch of a button or the move of a joystick—if the function for doing so were correct, but took 10 seconds, players would not be happy. Similarly, a search on Google that sifts through terabytes of data must also be fast.<label for="sn-0" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-0" class="margin-toggle"/><span class="sidenote"> Check it out: each search you do on Google reports how many results were found in how many fractions of a second.</span> In this section, we will discuss how to write tests for efficiency of functions.</p>
|
||||
<!-- Even more surprising than performance constraints, the performance of the tests themselves are incredibly important in well-designed software projects.
|
||||
The test suite itself can be quite large, and each time a developer fixes a bug (hopefully) or adds a new feature the tests must be re-run.
|
||||
The faster the test suite runs, the better software can reach its internal (e.g., alpha, beta) and external release deadlines.
|
||||
In this section, we will discuss how to setup tests for performance constraints for both functions and tests (which, coincidentally, are also functions). -->
|
||||
<h2 id="an-efficiency-test">An efficiency test</h2>
|
||||
<p>Earlier we saw how to use the <code>timeit</code> module to measure the time taken to execute a piece of Python code. Let’s see how we might setup a performance constraint using <code>timeit</code> and our implementation of <code>is_prime</code>:</p>
|
||||
<div class="sourceCode" id="cb1"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1"></a><span class="im">from</span> math <span class="im">import</span> floor, sqrt</span>
|
||||
<span id="cb1-2"><a href="#cb1-2"></a><span class="im">from</span> timeit <span class="im">import</span> timeit</span>
|
||||
<span id="cb1-3"><a href="#cb1-3"></a></span>
|
||||
<span id="cb1-4"><a href="#cb1-4"></a></span>
|
||||
<span id="cb1-5"><a href="#cb1-5"></a><span class="kw">def</span> is_prime(p: <span class="bu">int</span>) <span class="op">-></span> <span class="bu">bool</span>:</span>
|
||||
<span id="cb1-6"><a href="#cb1-6"></a> <span class="co">"""Return whether p is prime."""</span></span>
|
||||
<span id="cb1-7"><a href="#cb1-7"></a> possible_divisors <span class="op">=</span> <span class="bu">range</span>(<span class="dv">2</span>, floor(sqrt(p)) <span class="op">+</span> <span class="dv">1</span>)</span>
|
||||
<span id="cb1-8"><a href="#cb1-8"></a> <span class="cf">return</span> (</span>
|
||||
<span id="cb1-9"><a href="#cb1-9"></a> p <span class="op">></span> <span class="dv">1</span> <span class="kw">and</span></span>
|
||||
<span id="cb1-10"><a href="#cb1-10"></a> <span class="bu">all</span>(<span class="kw">not</span> p <span class="op">%</span> d <span class="op">==</span> <span class="dv">0</span> <span class="cf">for</span> d <span class="kw">in</span> possible_divisors)</span>
|
||||
<span id="cb1-11"><a href="#cb1-11"></a> )</span>
|
||||
<span id="cb1-12"><a href="#cb1-12"></a></span>
|
||||
<span id="cb1-13"><a href="#cb1-13"></a></span>
|
||||
<span id="cb1-14"><a href="#cb1-14"></a><span class="kw">def</span> test_is_prime_performance() <span class="op">-></span> <span class="va">None</span>:</span>
|
||||
<span id="cb1-15"><a href="#cb1-15"></a> <span class="co">"""Test the efficiency of is_prime."""</span></span>
|
||||
<span id="cb1-16"><a href="#cb1-16"></a> numbers_to_test <span class="op">=</span> <span class="bu">range</span>(<span class="dv">2</span>, <span class="dv">1000</span>)</span>
|
||||
<span id="cb1-17"><a href="#cb1-17"></a> <span class="cf">for</span> number <span class="kw">in</span> numbers_to_test:</span>
|
||||
<span id="cb1-18"><a href="#cb1-18"></a> time <span class="op">=</span> timeit(<span class="ss">f'is_prime(</span><span class="sc">{</span>number<span class="sc">}</span><span class="ss">)'</span>, number<span class="op">=</span><span class="dv">100</span>, <span class="bu">globals</span><span class="op">=</span><span class="bu">globals</span>())</span>
|
||||
<span id="cb1-19"><a href="#cb1-19"></a> <span class="cf">assert</span> time <span class="op"><</span> <span class="fl">0.001</span>, <span class="st">'Failed performance constraint of 0.001s.'</span></span></code></pre></div>
|
||||
<p>There are several issues here that we need to keep in mind. The performance constraint of 0.001 seconds is for the total runtime of 100 calls to <code>is_prime</code> for only one number in <code>numbers_to_test</code> (there will be as many assertions as there are elements in <code>numbers_to_test</code>). Where did the argument <code>number=100</code> come from? Should it be more or less? An important thing to remember is a computer system is not at all like a science experiment you would setup in a chemistry or biology lab. There are too many external factors (i.e., background processes being run) that can impact the results. To avoid this issue, several samples of an experiment (i.e., measurements of time) need to be taken. The field of statistics can help inform us on whether or not 100 samples is sufficient for this test.</p>
|
||||
<p>Next, where did <code>0.001</code> seconds come from? The number is most certainly arbitrary in this example. Computer systems are very different from one another, in terms of both hardware and software. While the assertions may hold for all <code>numbers_to_test</code> on one computer, they may not hold on another. The <code>0.001</code> seconds may be tuned over time in the testing suite. Or it can help identify the minimum hardware requirements for running a piece of software.</p>
|
||||
<p>While it is easy to write the Python code that checks for performance, coming up with the actual parameters (number of function calls, inputs to the function, total acceptable runtime) is quite challenging, and often domain-dependent. For example, in user interfaces, a great deal of research has gone into how fast actions should be; a so-called “instantaneous” action in a user interface should complete in <a href="https://www.nngroup.com/articles/response-times-3-important-limits/">0.1 seconds</a>. Other domains, such as embedded systems, have a series of functions that must meet hard deadlines in order for the computer system to function properly (e.g., in a spaceship).</p>
|
||||
<p>But what about domains where there are no guidelines or standards? Runtime constraints that are tuned over time can still be useful in discovering changes in program efficiency due to bug fixes or new features. When a code change causes an efficiency test to fail, the programmers can decide whether to the efficiency constraint or explore alternative code changes. Without efficiency tests in place, the change in performance might not have been found until it impacted a real user of the software!</p>
|
||||
<!-- ## Timing Out on Tests
|
||||
|
||||
The result of a test can be unpredictable.
|
||||
So far, we have assumed that a test can either pass or fail.
|
||||
But what if a test never ends?
|
||||
For example, a specific set of inputs could cause the function being tested to enter an infinite loop.
|
||||
In this scenario, the test does not know that the function is in an infinite loop, and so cannot raise an assertion error.
|
||||
In a less extreme example, a recent change in the code has increased the complexity of the function from logarithmic to quadratic.
|
||||
An large input that was tested earlier in seconds could now be taking minutes.
|
||||
|
||||
Both of the examples above motivate the need for a maximum time that each test should take.
|
||||
When the maximum time is exceeded, the test is said to have *timed out*.
|
||||
You may have seen this type of error before while browsing the internet.
|
||||
Sometimes, a server is unable to process a request within a reasonable amount of time, and your browser displays an error such as "connection timed out".
|
||||
Setting up timeouts for our tests is useful to ensure that all tests will complete.
|
||||
|
||||
We can add timeouts to our functions in pytest through the `timeout_decorator` Python module.
|
||||
The module provides access to a decorator: `@timeout`.
|
||||
We have seen decorators before when we introduced `@dataclass` before defining a new `class`.
|
||||
In this scenario, we place the `@timeout` decorator before defining a new test function:
|
||||
|
||||
```python
|
||||
import timeout_decorator
|
||||
|
||||
@timeout_decorator.timeout(1)
|
||||
def test_is_prime_large_number() -> None:
|
||||
assert is_prime(32443) is True
|
||||
```
|
||||
|
||||
Much like with performance constraints, we must specify how long (in seconds) a test should take.
|
||||
In this scenario, we timeout after `1` second.
|
||||
The decision on how many seconds is, similarly, a tuning exercise.
|
||||
To give you a very relevant example, your undergraduate programming classes choose timeouts based on a few parameters.
|
||||
How generous do the professors want to be with inefficient implementations?
|
||||
How much of a load will the test suite put on the servers?
|
||||
How long can the professors wait before getting the grades back?
|
||||
You can imagine that some submissions may timeout on several tests---if a 30 minute timeout is used, then a single submission could take several hours to test! -->
|
||||
</section>
|
||||
<footer>
|
||||
<a href="https://www.teach.cs.toronto.edu/~csc110y/fall/notes/">CSC110 Course Notes Home</a>
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user