- {
- “cells”: [
- {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“# Importing Jupyter Notebooks as Modules”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“It is a common problem that people want to import code from Jupyter Notebooks.n”, “This is made difficult by the fact that Notebooks are not plain Python files,n”, “and thus cannot be imported by the regular Python machinery.n”, “n”, “Fortunately, Python provides some fairly sophisticated [hooks](https://www.python.org/dev/peps/pep-0302/) into the import machinery,n”, “so we can actually make Jupyter notebooks importable without much difficulty,n”, “and only using public APIs.”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“import io, os, sys, types”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“from IPython import get_ipythonn”, “from nbformat import readn”, “from IPython.core.interactiveshell import InteractiveShell”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“Import hooks typically take the form of two objects:n”, “n”, “1. a Module Loader, which takes a module name (e.g. ‘IPython.display’), and returns a Modulen”, “2. a Module Finder, which figures out whether a module might exist, and tells Python what Loader to use”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“def find_notebook(fullname, path=None):n”, ” """find a notebook, given its fully qualified name and an optional pathn”, ” n”, ” This turns "foo.bar" into "foo/bar.ipynb"n”, ” and tries turning "Foo_Bar" into "Foo Bar" if Foo_Barn”, ” does not exist.n”, ” """n”, ” name = fullname.rsplit(‘.’, 1)[-1]n”, ” if not path:n”, ” path = [‘’]n”, ” for d in path:n”, ” nb_path = os.path.join(d, name + ".ipynb")n”, ” if os.path.isfile(nb_path):n”, ” return nb_pathn”, ” # let import Notebook_Name find "Notebook Name.ipynb"n”, ” nb_path = nb_path.replace("_", " ")n”, ” if os.path.isfile(nb_path):n”, ” return nb_pathn”, ” “
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“## Notebook Loader”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“Here we have our Notebook Loader.n”, “It’s actually quite simple - once we figure out the filename of the module,n”, “all it does is:n”, “n”, “1. load the notebook document into memoryn”, “2. create an empty Modulen”, “3. execute every cell in the Module namespacen”, “n”, “Since IPython cells can have extended syntax,n”, “the IPython transform is applied to turn each of these cells into their pure-Python counterparts before executing them.n”, “If all of your notebook cells are pure-Python,n”, “this step is unnecessary.”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“class NotebookLoader(object):n”, ” """Module Loader for Jupyter Notebooks"""n”, ” def __init__(self, path=None):n”, ” self.shell = InteractiveShell.instance()n”, ” self.path = pathn”, ” n”, ” def load_module(self, fullname):n”, ” """import a notebook as a module"""n”, ” path = find_notebook(fullname, self.path)n”, ” n”, ” print ("importing Jupyter notebook from %s" % path)n”, ” n”, ” # load the notebook objectn”, ” with io.open(path, ‘r’, encoding=’utf-8’) as f:n”, ” nb = read(f, 4)n”, ” n”, ” n”, ” # create the module and add it to sys.modulesn”, ” # if name in sys.modules:n”, ” # return sys.modules[name]n”, ” mod = types.ModuleType(fullname)n”, ” mod.__file__ = pathn”, ” mod.__loader__ = selfn”, ” mod.__dict__[‘get_ipython’] = get_ipythonn”, ” sys.modules[fullname] = modn”, ” n”, ” # extra work to ensure that magics that would affect the user_nsn”, ” # actually affect the notebook module’s nsn”, ” save_user_ns = self.shell.user_nsn”, ” self.shell.user_ns = mod.__dict__n”, ” n”, ” try:n”, ” for cell in nb.cells:n”, ” if cell.cell_type == ‘code’:n”, ” # transform the input to executable Pythonn”, ” code = self.shell.input_transformer_manager.transform_cell(cell.source)n”, ” # run the code in themodulen”, ” exec(code, mod.__dict__)n”, ” finally:n”, ” self.shell.user_ns = save_user_nsn”, ” return modn”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“## The Module Finder”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“The finder is a simple object that tells you whether a name can be imported,n”, “and returns the appropriate loader.n”, “All this one does is check, when you do:n”, “n”, “
`python\n", "import mynotebook\n", "`n”, “n”, “it checks whether mynotebook.ipynb exists.n”, “If a notebook is found, then it returns a NotebookLoader.n”, “n”, “Any extra logic is just for resolving paths within packages.”]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“class NotebookFinder(object):n”, ” """Module finder that locates Jupyter Notebooks"""n”, ” def __init__(self):n”, ” self.loaders = {}n”, ” n”, ” def find_module(self, fullname, path=None):n”, ” nb_path = find_notebook(fullname, path)n”, ” if not nb_path:n”, ” returnn”, ” n”, ” key = pathn”, ” if path:n”, ” # lists aren’t hashablen”, ” key = os.path.sep.join(path)n”, ” n”, ” if key not in self.loaders:n”, ” self.loaders[key] = NotebookLoader(path)n”, ” return self.loaders[key]n”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“## Register the hook”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“Now we register the NotebookFinder with sys.meta_path”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“sys.meta_path.append(NotebookFinder())”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“After this point, my notebooks should be importable.n”, “n”, “Let’s look at what we have in the CWD:”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“ls nbpackage”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“So I should be able to import nbpackage.mynotebook.”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“import nbpackage.mynotebook”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“### Aside: displaying notebooks”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“Here is some simple code to display the contents of a notebookn”, “with syntax highlighting, etc.”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“from pygments import highlightn”, “from pygments.lexers import PythonLexern”, “from pygments.formatters import HtmlFormattern”, “n”, “from IPython.display import display, HTMLn”, “n”, “formatter = HtmlFormatter()n”, “lexer = PythonLexer()n”, “n”, “# publish the CSS for pygments highlightingn”, “display(HTML("""n”, “<style type=’text/css’>n”, “%sn”, “</style>n”, “""" % formatter.get_style_defs()n”, “))”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“def show_notebook(fname):n”, ” """display a short summary of the cells of a notebook"""n”, ” with io.open(fname, ‘r’, encoding=’utf-8’) as f:n”, ” nb = read(f, 4)n”, ” html = []n”, ” for cell in nb.cells:n”, ” html.append("<h4>%s cell</h4>" % cell.cell_type)n”, ” if cell.cell_type == ‘code’:n”, ” html.append(highlight(cell.source, lexer, formatter))n”, ” else:n”, ” html.append("<pre>%s</pre>" % cell.source)n”, ” display(HTML(’\n’.join(html)))n”, “n”, “show_notebook(os.path.join("nbpackage", "mynotebook.ipynb"))”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“So my notebook has some code cells,n”, “one of which contains some IPython syntax.n”, “n”, “Let’s see what happens when we import it”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“from nbpackage import mynotebook”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“Hooray, it imported! Does it work?”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“mynotebook.foo()”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“Hooray again!n”, “n”, “Even the function that contains IPython syntax works:”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“mynotebook.has_ip_syntax()”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“## Notebooks in packages”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“We also have a notebook inside the nb package,n”, “so let’s make sure that works as well.”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“ls nbpackage/nbs”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“Note that the __init__.py is necessary for nb to be considered a package,n”, “just like usual.”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“show_notebook(os.path.join("nbpackage", "nbs", "other.ipynb"))”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“from nbpackage.nbs import othern”, “other.bar(5)”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“So now we have importable notebooks, from both the local directory and inside packages.n”, “n”, “I can even put a notebook inside IPython, to further demonstrate that this is working properly:”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“import shutiln”, “from IPython.paths import get_ipython_package_dirn”, “n”, “utils = os.path.join(get_ipython_package_dir(), ‘utils’)n”, “shutil.copy(os.path.join("nbpackage", "mynotebook.ipynb"),n”, ” os.path.join(utils, "inside_ipython.ipynb")n”, “)”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“and import the notebook from IPython.utils”
]
}, {
“cell_type”: “code”, “execution_count”: null, “metadata”: {
“collapsed”: false
}, “outputs”: [], “source”: [
“from IPython.utils import inside_ipythonn”, “inside_ipython.whatsmyname()”
]
}, {
“cell_type”: “markdown”, “metadata”: {}, “source”: [
“This approach can even import functions and classes that are defined in a notebook using the %%cython magic.”
]
}
], “metadata”: {
“gist_id”: “6011986”, “nbsphinx”: {
“execute”: “never”
}, “kernelspec”: {
“display_name”: “Python 3”, “language”: “python”, “name”: “python3”
}, “language_info”: {
- “codemirror_mode”: {
“name”: “ipython”, “version”: 3
}, “file_extension”: “.py”, “mimetype”: “text/x-python”, “name”: “python”, “nbconvert_exporter”: “python”, “pygments_lexer”: “ipython3”, “version”: “3.5.1+”
}
}, “nbformat”: 4, “nbformat_minor”: 0
}