All-optical machine learning using diffractive deep neural networks
All-optical deep learning
Deep learning uses multilayered artificial neural networks to learn digitally from large datasets. It then performs advanced identification and classification tasks. To date, these multilayered neural networks have been implemented on a computer. Lin et al. demonstrate all-optical machine learning that uses passive optical components that can be patterned and fabricated with 3D-printing. Their hardware approach comprises stacked layers of diffractive optical elements analogous to an artificial neural network that can be trained to execute complex functions at the speed of light.
Science, this issue p. 1004
Abstract
Deep learning has been transforming our ability to execute advanced inference tasks using computers. Here we introduce a physical mechanism to perform machine learning by demonstrating an all-optical diffractive deep neural network (D2NN) architecture that can implement various functions following the deep learning–based design of passive diffractive layers that work collectively. We created 3D-printed D2NNs that implement classification of images of handwritten digits and fashion products, as well as the function of an imaging lens at a terahertz spectrum. Our all-optical deep learning framework can perform, at the speed of light, various complex functions that computer-based neural networks can execute; will find applications in all-optical image analysis, feature detection, and object classification; and will also enable new camera designs and optical components that perform distinctive tasks using D2NNs.
Get full access to this article
View all available purchase options and get full access to this article.
Already a Subscriber?Sign In
Supplementary Material
Summary
Materials and Methods
Figs. S1 to S16
Resources
References and Notes
1
Y. LeCun, Y. Bengio, G. Hinton, Deep learning. Nature 521, 436–444 (2015).
2
G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. W. M. van der Laak, B. van Ginneken, C. I. Sánchez, A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017).
3
A. Graves, A. Mohamed, G. Hinton, in Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (IEEE, 2013), pp. 6645–6649.
4
K. Cho et al., Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078 [cs.CL] (3 June 2014).
5
A. Krizhevsky, I. Sutskever, G. E. Hinton, in Advances in Neural Information Processing Systems 25 (NIPS 2012), F. Pereira, C. J. C. Burges, L. Bottou, K. Q. Weinberger, Eds. (Curran Associates, 2012), pp. 1097–1105.
6
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, D. Hassabis, Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).
7
U. S. Kamilov, I. N. Papadopoulos, M. H. Shoreh, A. Goy, C. Vonesch, M. Unser, D. Psaltis, Learning approach to optical tomography. Optica 2, 517 (2015).
8
Y. Rivenson, Z. Göröcs, H. Günaydin, Y. Zhang, H. Wang, A. Ozcan, Deep learning microscopy. Optica 4, 1437 (2017).
9
K. H. Jin, M. T. McCann, E. Froustey, M. Unser, Deep convolutional neural network for inverse problems in imaging. IEEE Trans. Image Process. 26, 4509–4522 (2017).
10
Y. Rivenson, Y. Zhang, H. Gunaydin, D. Teng, A. Ozcan, Phase recovery and holographic image reconstruction using deep learning in neural networks. Light Sci. Appl. 7, 17141 (2018).
11
A. Sinha, J. Lee, S. Li, G. Barbastathis, Lensless computational imaging through deep learning. Optica 4, 1117 (2017).
12
K. Hammernik, T. Klatzer, E. Kobler, M. P. Recht, D. K. Sodickson, T. Pock, F. Knoll, Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson. Med. 79, 3055–3071 (2018).
13
Y. Rivenson, H. Ceylan Koydemir, H. Wang, Z. Wei, Z. Ren, H. Günaydın, Y. Zhang, Z. Göröcs, K. Liang, D. Tseng, A. Ozcan, Deep learning enhanced mobile-phone microscopy. ACS Photonics 5, 2354–2364 (2018).
14
Materials and methods are available as supplementary materials.
15
Y. Lecun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998).
16
D. Ciregan, U. Meier, J. Schmidhuber, in Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2012), pp. 3642–3649.
17
C.-Y. Lee, P. W. Gallagher, Z. Tu, in Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (AISTATS, 2016), pp. 464–472.
18
M. A. Ranzato, C. Poultney, S. Chopra, Y. LeCun, in Advances in Neural Information Processing Systems 20 (NIPS 2007), J. C. Platt, D. Koller, Y. Singer, S. T. Roweis, Eds. (Curran Associates, 2007), pp. 1137–1144.
19
H. Xiao, K. Rasul, R. Vollgraf, Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv:1708.07747 [cs.LG] (25 August 2017).
20
S. Zeng, B. Zhang, J. Gou, Deep Collaborative Weight-based Classification. arXiv:1802.07589 [cs.CV] (21 February 2018).
21
ImageNet database, www.image-net.org.
22
Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, M. Soljačić, Deep learning with coherent nanophotonic circuits. Nat. Photonics 11, 441–446 (2017).
23
D. Psaltis, D. Brady, X.-G. Gu, S. Lin, Holography in artificial neural networks. Nature 343, 325–330 (1990).
24
K. H. Wagner, in Proceedings of Frontiers in Optics 2017 (OSA, 2017), FW2C.1.
25
B. J. Shastri et al., Principles of neuromorphic photonics. arXiv:1801.00016 [cs.ET] (29 December 2017).
26
M. Hermans, M. Burm, T. Van Vaerenbergh, J. Dambre, P. Bienstman, Trainable hardware for dynamical computing using error backpropagation through physical media. Nat. Commun. 6, 6729 (2015).
27
D. Brunner, M. C. Soriano, C. R. Mirasso, I. Fischer, Parallel photonic information processing at gigabyte per second data rates using transient states. Nat. Commun. 4, 1364 (2013).
28
A. Greenbaum, W. Luo, T.-W. Su, Z. Göröcs, L. Xue, S. O. Isikman, A. F. Coskun, O. Mudanyali, A. Ozcan, Imaging without lenses: Achievements and remaining challenges of wide-field on-chip microscopy. Nat. Methods 9, 889–895 (2012).
29
A. Ozcan, E. McLeod, Lensless imaging and sensing. Annu. Rev. Biomed. Eng. 18, 77–102 (2016).
30
M. Emons, K. Obata, T. Binhammer, A. Ovsianikov, B. N. Chichkov, U. Morgner, Two-photon polymerization technique with sub-50 nm resolution by sub-10 fs laser pulses. Opt. Mater. Express 2, 942–947 (2012).
31
D. P. Kingma, J. Ba, Adam: A method for stochastic optimization. arXiv:1412.6980 [cs.LG] (22 December 2014).
32
M. Kazhdan, H. Hoppe, Screened poisson surface reconstruction. ACM Trans. Graph. 32, 1–13 (2013).
33
MeshLab, www.meshlab.net.
34
P. Cignoni, M. Callieri, M. Corsini, M. Dellepiane, F. Ganovelli, G. Ranzuglia, “MeshLab: An open-source mesh processing tool” in Sixth Eurographics Italian Chapter Conference (2008), pp. 129–136.
35
J. W. Goodman, Introduction to Fourier Optics (Roberts and Company, 2005).
36
C. Trabelsi et al., Deep complex networks. arXiv:1705.09792 [cs.NE] (27 May 2017).
37
Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).
38
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014).
39
V. Bianchi, T. Carey, L. Viti, L. Li, E. H. Linfield, A. G. Davies, A. Tredicucci, D. Yoon, P. G. Karagiannidis, L. Lombardi, F. Tomarchio, A. C. Ferrari, F. Torrisi, M. S. Vitiello, Terahertz saturable absorbers from liquid phase exfoliation of graphite. Nat. Commun. 8, 15763 (2017).
40
A. Marini, J. D. Cox, F. J. García de Abajo, Theory of graphene saturable absorption. Phys. Rev. B 95, 125408 (2017).
41
X. Yin, T. Feng, Z. Liang, J. Li, Artificial Kerr-type medium using metamaterials. Opt. Express 20, 8543–8550 (2012).
42
Y. Xiao, H. Qian, Z. Liu, Nonlinear metasurface based on giant optical kerr response of gold quantum wells. ACS Photonics 5, 1654–1659 (2018).
43
N. Yu, F. Capasso, Flat optics with designer metasurfaces. Nat. Mater. 13, 139–150 (2014).
44
M. Khorasaninejad, W. T. Chen, R. C. Devlin, J. Oh, A. Y. Zhu, F. Capasso, Metalenses at visible wavelengths: Diffraction-limited focusing and subwavelength resolution imaging. Science 352, 1190–1194 (2016).
45
A. V. Kildishev, A. Boltasseva, V. M. Shalaev, Planar photonics with metasurfaces. Science 339, 1232009 (2013).
46
D. Grischkowsky, S. Keiding, M. Van Exter, C. Fattinger, Far-infrared time-domain spectroscopy with terahertz beams of dielectrics and semiconductors. J. Opt. Soc. Am. B 7, 2006–2015 (1990).
Information & Authors
Information
Published In
Science
Volume 361 | Issue 6406
7 September 2018
7 September 2018
Copyright
Copyright © 2018 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works.
This is an article distributed under the terms of the Science Journals Default License.
Article versions
You are viewing the most recent version of this article.
Submission history
Received: 6 April 2018
Accepted: 12 July 2018
Published in print: 7 September 2018
Acknowledgments
We thank D. Mengu, Z. Wei, X. Wang, and Y. Xiao of UCLA for assistance with coding. Funding: The Ozcan Group at UCLA acknowledges the support of the National Science Foundation and the Howard Hughes Medical Institute. Author contributions: A.O., X.L., and Y.R. conceived of the research; X.L., N.T.Y., Y.L., Y.R., M.V., and M.J. contributed to the experiments; X.L., N.T.Y., M.V., and Y.R. processed the data; A.O., X.L., M.V., N.T.Y., Y.R., Y.L., and M.J. prepared the manuscript; and A.O. initiated and supervised the research. Competing interests: A.O., X.L., and Y.R. are inventors of a patent application on D2NNs. Data and materials availability: All data and methods are present in the main text and supplementary materials.
Authors
Funding Information
Metrics & Citations
Metrics
Article Usage
Altmetrics
Citations
Export citation
Select the format you want to export the citation of this publication.
Cited by
- Machine Learning Approach for On-Demand Rapid Constructing Metasurface, IEEE Journal of Selected Topics in Quantum Electronics, 28, 4, (1-9), (2022).https://doi.org/10.1109/JSTQE.2021.3083565
- Diffractive Deep Neural Network for Optical Orbital Angular Momentum Multiplexing and Demultiplexing, IEEE Journal of Selected Topics in Quantum Electronics, 28, 4, (1-11), (2022).https://doi.org/10.1109/JSTQE.2021.3077907
- Optical secret sharing with cascaded metasurface holography, Science Advances, 7, 16, (2021)./doi/10.1126/sciadv.abf9718
- Deep learning in nano-photonics: inverse design and beyond, Photonics Research, 9, 5, (B182), (2021).https://doi.org/10.1364/PRJ.415960
- Spatial Mode Correction of Single Photons Using Machine Learning, Advanced Quantum Technologies, 4, 3, (2000103), (2021).https://doi.org/10.1002/qute.202000103
- undefined, Integrated Optics: Design, Devices, Systems and Applications VI, (1), (2021).https://doi.org/10.1117/12.2588733
- Addressing limited weight resolution in a fully optical neuromorphic reservoir computing readout, Scientific Reports, 11, 1, (2021).https://doi.org/10.1038/s41598-021-82720-4
- Arbitrary linear transformations for photons in the frequency synthetic dimension, Nature Communications, 12, 1, (2021).https://doi.org/10.1038/s41467-021-22670-7
- Training Deep Photonic Convolutional Neural Networks With Sinusoidal Activations, IEEE Transactions on Emerging Topics in Computational Intelligence, 5, 3, (384-393), (2021).https://doi.org/10.1109/TETCI.2019.2923001
- Interfacing photonics with artificial intelligence: an innovative design strategy for photonic structures and devices based on artificial neural networks, Photonics Research, 9, 4, (B135), (2021).https://doi.org/10.1364/PRJ.417693
- See more
Loading...
View Options
Get Access
Log in to view the full text
AAAS login provides access to Science for AAAS members, and access to other journals in the Science family to users who have purchased individual subscriptions.
- Become a AAAS Member
- Activate your Account
- Purchase Access to Other Journals in the Science Family
- Account Help
Log in via OpenAthens.
Log in via Shibboleth.
More options
Register for free to read this article
As a service to the community, this article is available for free. Login or register for free to read this article.
Buy a single issue of Science for just $15 USD.
View options
PDF format
Download this article as a PDF file
Download PDF
A "diffractive deep neural network" is NOT an all-optical implementation of a conventional deep neural network
Here Lin et al. report a remarkable proposal that employs a passive, strictly linear optical setup to perform pattern classifications. But readers are strongly advised to draw a clear distinction between the so-called "diffractive deep neural network" (D2NN) and an all-optical implementation of a deep neural network (DNN) in the canonical sense. While the purported D2NN is devoid of any substantially nonlinear signal processing in the middle (hidden) layers, a conventional DNN incorporates nonlinear activations in its middle (hidden) layers and derives powerful computational advantages from them.
Interested readers are referred to H. Wei et al., "Comment on 'All-optical machine learning using diffractive deep neural networks'," arXiv:1809.08360 (https://arxiv.org/abs/1809.08360) for detailed discussions, where it has also been rigorously proved that any nonlinearity present or introduced at the output layer of a D2NN or afterward won't be able to boost the pattern discrimination power to beyond the classical Euclidean distance-based algorithms.