Extending Theano

This advanced tutorial is for users who want to extend Theano with new Types, new Operations (Ops), and new graph optimizations.

Along the way, it also introduces many aspects of how Theano works, so it is also good for you if you are interested in getting more under the hood with Theano itself.

Before tackling this more advanced presentation, it is highly recommended to read the introductory Tutorial.

The first few pages will walk you through the definition of a new Type, double, and a basic arithmetic operations on that Type. We will start by defining them using a Python implementation and then we will add a C implementation.

• Writing an Op to work on an ndarray in C
  • Writing an Optimization
  • Test the optimization
• Overview of the compilation pipeline
  • Definition of the computation graph
  • Compilation of the computation graph
    • Step 1 - Create a FunctionGraph
    • Step 2 - Execute main Optimizer
    • Step 3 - Execute linker to obtain a thunk
    • Step 4 - Wrap the thunk in a pretty package
• Theano vs. C
• Graph Structures
  • An explicit example
    • Automatic wrapping
  • Graph Structures
    • Apply
      • Op
      • Type
      • Variable
        • Constant
  • Graph Structures Extension
• Making the double type
  • Type’s contract
    • Additional definitions
      • transfer()
  • Defining double
  • Untangling some concepts
  • Final version
• Making arithmetic Ops on double
  • Op’s contract
  • Optional methods or attributes
  • Gradient
  • Defining an Op: mul
  • Trying out our new Op
    • Automatic Constant Wrapping
  • Final version
• Views and inplace operations
  • Views
  • Inplace operations
  • Destructive Operations
  • Inplace optimization and DebugMode
• Implementing some specific Ops
  • Scalar/Elemwise/Reduction Ops
  • SciPy Ops
  • Sparse Ops
    • Sparse Gradient
    • Sparse C code
    • Sparse Tests
  • Random distribution
  • OpenMP Ops
  • Numba Ops
  • Alternate Theano Types
• Implementing double in C
  • How does it work?
  • What needs to be defined
  • Defining the methods
  • What the generated C will look like
  • Final version
  • DeepCopyOp
  • ViewOp
  • Shape and Shape_i
• Implementing the arithmetic Ops in C
  • How does it work?
  • What needs to be defined
  • Defining the methods
  • Final version
• Using Op params
  • The params type
    • Defining a params type
  • Registering the params with your Op
  • Signature changes from having params
  • A simple example
  • A more complex example
• Graph optimization
  • Global and local optimizations
    • Global optimization
    • Local optimization
  • One simplification rule
    • Global optimization
    • Local optimization
      • OpSub, OpRemove, PatternSub
  • The optimization database (optdb)
    • Definition of optdb
    • Query
    • Examples
    • Registering an Optimizer
    • Registering a LocalOptimizer
    • optdb structure
    • Navigator
  • Profiling Theano function compilation
    • Detailed profiling of Theano optimizer
• Tips
  • Reusing outputs
  • Don’t define new Ops unless you have to
  • Use Theano’s high order Ops when applicable
  • Op Checklist
• Unit Testing
  • Unittest Primer
    • How to Run Unit Tests ?
      • theano-nose
      • Using unittest module
    • Folder Layout
  • How to Write a Unittest
    • Test Cases and Methods
    • Checking for correctness
    • Checking for errors
    • Test Cases and Theano Modes
    • Using Random Values in Test Cases
  • Creating an Op UnitTest
    • Validating the Gradient
  • makeTester and makeBroadcastTester
• Extending Theano: FAQ and Troubleshooting
  • I wrote a new Op/Type, and weird stuff is happening...
  • I wrote a new optimization, but it’s not getting used...
  • I wrote a new optimization, and it changed my results even though I’m pretty sure it is correct.