MIPT SciTech Club

AI courses Machine learning & AI http://cs229.stanford.edu/syllabus.html Machine learning https://web.stanford.edu/class/cs229t/ Statistical learning theory https://cs.stanford.edu/~ermon/cs228/index.html Probabilistic graphical models https://web.stanford.edu/class/cs221/ AI: principles and techniques Com...

Одобряем 🙂

Eric Gaussier (CUEF de Grenoble - Université Grenoble Alpes) about "Semantic annotation in the biomedical domain: large scale classification and BioASQ"

- Eric Gaussier -> http://ama.liglab.fr/~gaussier/
- Presentation file with similar content ->http://statlearn.sfds.asso.fr/wp-content/uploads/2015/04/3-Gaussier.pdf
- BioASQ Challenge (A challenge on large-scale
biomedical semantic indexing and question answering) -> http://bioasq.org/participate/challenges

Approaches for classification in large-scale taxonomies:
* Hierarchical
* * Top-down (e.g. “SVMs Classification with A Very Large-scale Taxonomy” [Liu et al., 2005, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.72.1506&rep=rep1&type=pdf], “Refined expert- s: improving classification in large taxonomies“ [Bennett and Nguyen, 2009, http://www.cs.cornell.edu/~nhnguyen/refined_experts.pdf])
* * Big-bang (e.g. “Hierarchical document categorization with support vector machines” [Cai and Hofman , 2004, https://dl.acm.org/citation.cfm?id=1031186], “Distribution-Calibrated Hierarchical Classification” [Dekel, 2009, https://papers.nips.cc/paper/3629-distribution-calibrated-hierarchical-classification.pdf], “Bayesian models for large-scale hierarchical classification” [Gopal et al., 2012, https://www.researchgate.net/publication/290780111_Bayesian_models_for_large-scale_hierarchical_classification])
* Flat (e.g. “Label Embedding Trees for Large Multi-Class Tasks“ [Bengio et al., 2010, https://papers.nips.cc/paper/4027-label-embedding-trees-for-large-multi-class-tasks.pdf])
* Mildly hierarchical (e.g. “Improving Hierarchical SVMs by Hierarchy Flattening and Lazy Classification” [Malik, 2009, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.628.5478&rep=rep1&type=pdf])

Source: https://mipt.ru/education/departments/fpmi/events/erik_gose_semanticheskoe_annotirovanie_i_voprosno_otvetnye_sistemy_bioasq

https://youtu.be/uHlYDJ8evfo

Eric Gaussier. Semantic annotation in the biomedical domain: large scale classification and BioASQ http://ama.liglab.fr/~gaussier/ https://mipt.ru/education/departments/fpmi/events/erik_gose_semanticheskoe_annotirovanie_i_voprosno_otvetnye_sistemy_bioasq

MIPT DL Club #14
Anton Karazeev about Evolution Strategies (ES) -- an alternative method of Reinforcement Learning (RL). Paper -> https://arxiv.org/abs/1703.03864

An interesting feature of this ES method is that it doesn’t require backpropagation of error and that this algorithm (ES) is highly parallelizable. The latter is confirmed by the following experiment: MuJoCo, 3D humanoid was trained [ES] for 10 minutes using 1440 CPUs and 80 machines, [A3C] for 10 hours using 32 A3C-workers on one machine (since A3C/TRPO are hard to parallelize due to the machines would have to have a high bandwidth to share data)

Thanks to Danila Doroshin for recommendation to read this interesting paper :)

- On OpenAI’s blog -> https://blog.openai.com/evolution-strategies/
- Jupyter Notebook on Andrej Karpathy’s GitHub -> https://github.com/karpathy/randomfun/blob/master/es.ipynb
- Insights into ES by Ilya Sutskever -> 53:40, https://youtu.be/9EN_HoEk3KY (thanks to http://t.me/higgsfield for the link)
- Trust Region Policy Optimization -> https://arxiv.org/abs/1502.05477

https://youtu.be/8jKC95KklT0

MIPT DL Club #14. Evolution Strategies

MIPT Bioinf Club #1
Daria Romanovskaia about Nature paper "Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning" (https://www.nature.com/articles/nbt.3300)

Authors of this paper proposed new method based on Deep Learning (DL) to predict binding sites of RNA- and DNA-binding proteins that are regulators of gene expression.

Method is based on Convolutional Neural Networks (CNN) that help to assign affinity score to every sequence.

As a result - proposed method is very accurate and it works much better than already existent programs for searching of biological motifs in DNA and RNA. A useful add-on is the visualization of maps of potentially pathogenic mutations.

https://youtu.be/_NagRpo_Eww

MIPT Bioinf Club #1. DeepBind

Michael Chertkov about "Science Application Informed Machine Learning" [ in Physics, application of to the problem of turbulence, controlling of energy networks in USA, etc.]

Abstract: Thanks to IT industry push, Machine Learning (ML) capabilities are in a phase of tremendous growth, and there is great opportunity to point these practically powerful tools toward modeling specific to applications, e.g. in natural and engineering sciences. The challenge is to incorporate domain expertise from traditional scientific discovery into next-generation ML models. We propose to develop new theoretical and algorithmic methodology that extends cutting-edge ML tools and merge them with application-specific knowledge stated in the form of constraints, symmetries, conservation laws, phenomenological assumptions and other examples of domain expertise regarding relevant degrees of freedom.

The emerging methodology is illustrated on the following four enabling examples:

1. Topology and Parameter Estimation in Power Grids [IEEE CONES 2018/ https://arxiv.org/abs/1710.10727]
2. Acceleration of Computational Fluid Dynamics with Deep Learning [APS/DFD2017 abstract + work in progress]
3. Learning Graphical Models [Science 2018/ https://arxiv.org/abs/1612.05024] and / https://arxiv.org/abs/1605.07252]
4. Renormalization of Tensor Networks (Graphical Models) [AISTATS 2018/ https://arxiv.org/abs/1801.01649 and 2018/ https://arxiv.org/abs/1803.05104]

Source: https://mipt.ru/education/departments/fpmi/events/matematicheskiy_kruzhok_shkoly

https://youtu.be/r1EImGv0NxE

Science Application Informed Machine Learning. Michael Chertkov Abstract: Thanks to IT industry push, Machine Learning (ML) capabilities are in a phase of tremendous growth, and there is great opportunity to point these p...

MIPT Q Club #2
Anton Karazeev about optical setups that can mimic the functionality of artificial neural networks (Optical Neural Networks) - paper [1], Nature, 2017.

The linear (and some nonlinear) transformations can be applied at the speed of light in optical setups. It's well known from physics that a lens performs Fourier transform without any energy consumption and consequently some matrix operations can be performed optically without consumption of energy.

These advantages in speed and energy consumption make optical neural networks (ONNs) fairly prospective field of research.

Authors [1] implemented nanophotonic circuit and classified spoken vowels with it (they "trained" Mach-Zender interferometer by changing the phase shifts of laser beam). According to the paper, the nonlinearity wasn't implemented optically but was calculated on classical computer. Perhaps the training phase and nonlinearity will be implemented inside the optical circuit in next versions.

[1] Deep learning with coherent nanophotonic circuits (https://www.nature.com/articles/nphoton.2017.93)
[2] Mach–Zehnder interferometer (https://en.wikipedia.org/wiki/Mach–Zehnder_interferometer)
[3] Computing by Means of Physics-Based Optical Neural Networks (https://arxiv.org/abs/1006.1434)

*OIU - Optical Interference Unit

https://youtu.be/bpkuyGXvbEU

MIPT Q Club #2. Optical Neural Networks

MIPT DL Club #13
Taras Khakhulin about "Breaking the Softmax Bottleneck: A High-Rank RNN Language Model" (https://arxiv.org/abs/1711.03953)

The problem of constructing the Language model was considered as a factorization of matrix. Authors showed that Softmax has a bottleneck which affects the expressive power of the model. Also they proposed to solve such a problem using the Mixture of Softmax.

Results mentioned in the paper are impressive. A lot of experiments were conducted and state-of-the-art results were achieved on a big number of datasets.

To conclude, even very strong RNN Language model's expressive power will be restricted because of a high rank representation of natural language.

https://youtu.be/IXI5EkCyVnA

MIPT DL Club #13. Breaking the Softmax Bottleneck

MIPT DL Club #12
Anton Karazeev on classical methods of keyphrases extraction from [biomedical] texts: statistical (TF-IDF) and graphical (TextRank).

Besides them, newly proposed methods were discussed: based on Information Theory (Kullback-Leibler divergence) using word2gauss algorithm.

- word2gauss (https://github.com/seomoz/word2gauss)
- [2015, ICLR] Word Representations via Gaussian Embedding (https://arxiv.org/abs/1412.6623)
- [2017] Multimodal Word Distributions (https://arxiv.org/abs/1704.08424)
- [2018] EmbedRank: Unsupervised Keyphrase Extraction using Sentence Embeddings (https://arxiv.org/abs/1801.04470)
- [2010] Keyword Extraction Using Word Co-occurrence (https://www.researchgate.net/publication/224179686_Ke..)

https://youtu.be/D9nzI0EO9ok

MIPT DL Club #12. Keyphrase extraction