Spark Package Management Inside Jupyter Notebooks
It would be nice to be able to manage spark packages (say from http://spark-packages.org/) inside a jupyter notebook. This is implemented in the IBM spark kernel using Line Magics. Specifically, this is done in the ibm spark kernal using the %AddDeps line magic (https://github.com/ibm-et/spark-kernel/wiki/List-of-Current-Magics-for-the-Spark-Kernel). It would be great to have a similar story in HDInsight.
I guess this would probably be done with something like Apache Toree.
'...Toree provides a well-defined mechanism to associate functionality with magics, and this is a useful point of extensibility of the system'
The link I posted for the line magics is messed up. Here is the real link: https://github.com/ibm-et/spark-kernel/wiki/List-of-Current-Magics-for-the-Spark-Kernel