Advertisement
GET OUR E-ALERTS
Report

All-optical machine learning using diffractive deep neural networks

Xing Lin https://orcid.org/0000-0002-5560-7286, Yair Rivenson, Nezih T. Yardimci https://orcid.org/0000-0002-3623-5835, Muhammed Veli, Yi Luo, Mona Jarrahi, and Aydogan Ozcan https://orcid.org/0000-0002-0717-683X [email protected]
Science • 26 Jul 2018 • Vol 361, Issue 6406 • pp. 1004-1008 • DOI: 10.1126/science.aat8084
get access

All-optical deep learning

Deep learning uses multilayered artificial neural networks to learn digitally from large datasets. It then performs advanced identification and classification tasks. To date, these multilayered neural networks have been implemented on a computer. Lin et al. demonstrate all-optical machine learning that uses passive optical components that can be patterned and fabricated with 3D-printing. Their hardware approach comprises stacked layers of diffractive optical elements analogous to an artificial neural network that can be trained to execute complex functions at the speed of light.
Science, this issue p. 1004

Abstract

Deep learning has been transforming our ability to execute advanced inference tasks using computers. Here we introduce a physical mechanism to perform machine learning by demonstrating an all-optical diffractive deep neural network (D2NN) architecture that can implement various functions following the deep learning–based design of passive diffractive layers that work collectively. We created 3D-printed D2NNs that implement classification of images of handwritten digits and fashion products, as well as the function of an imaging lens at a terahertz spectrum. Our all-optical deep learning framework can perform, at the speed of light, various complex functions that computer-based neural networks can execute; will find applications in all-optical image analysis, feature detection, and object classification; and will also enable new camera designs and optical components that perform distinctive tasks using D2NNs.
Get full access to this article

View all available purchase options and get full access to this article.

Get Access
Already a Subscriber?

Supplementary Material

Summary

Materials and Methods
Figs. S1 to S16
References (31–46)

Resources

File (aat8084-lin-sm-rev-3.pdf)
Download
File (aat8084-lin-sm.pdf)
Download

References and Notes

1
Y. LeCun, Y. Bengio, G. Hinton, Deep learning. Nature 521, 436–444 (2015).
2
G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. W. M. van der Laak, B. van Ginneken, C. I. Sánchez, A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017).
3
A. Graves, A. Mohamed, G. Hinton, in Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (IEEE, 2013), pp. 6645–6649.
4
K. Cho et al., Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078 [cs.CL] (3 June 2014).
5
A. Krizhevsky, I. Sutskever, G. E. Hinton, in Advances in Neural Information Processing Systems 25 (NIPS 2012), F. Pereira, C. J. C. Burges, L. Bottou, K. Q. Weinberger, Eds. (Curran Associates, 2012), pp. 1097–1105.
6
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, D. Hassabis, Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).
7
U. S. Kamilov, I. N. Papadopoulos, M. H. Shoreh, A. Goy, C. Vonesch, M. Unser, D. Psaltis, Learning approach to optical tomography. Optica 2, 517 (2015).
8
Y. Rivenson, Z. Göröcs, H. Günaydin, Y. Zhang, H. Wang, A. Ozcan, Deep learning microscopy. Optica 4, 1437 (2017).
9
K. H. Jin, M. T. McCann, E. Froustey, M. Unser, Deep convolutional neural network for inverse problems in imaging. IEEE Trans. Image Process. 26, 4509–4522 (2017).
10
Y. Rivenson, Y. Zhang, H. Gunaydin, D. Teng, A. Ozcan, Phase recovery and holographic image reconstruction using deep learning in neural networks. Light Sci. Appl. 7, 17141 (2018).
11
A. Sinha, J. Lee, S. Li, G. Barbastathis, Lensless computational imaging through deep learning. Optica 4, 1117 (2017).
12
K. Hammernik, T. Klatzer, E. Kobler, M. P. Recht, D. K. Sodickson, T. Pock, F. Knoll, Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson. Med. 79, 3055–3071 (2018).
13
Y. Rivenson, H. Ceylan Koydemir, H. Wang, Z. Wei, Z. Ren, H. Günaydın, Y. Zhang, Z. Göröcs, K. Liang, D. Tseng, A. Ozcan, Deep learning enhanced mobile-phone microscopy. ACS Photonics 5, 2354–2364 (2018).
14
Materials and methods are available as supplementary materials.
15
Y. Lecun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998).
16
D. Ciregan, U. Meier, J. Schmidhuber, in Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2012), pp. 3642–3649.
17
C.-Y. Lee, P. W. Gallagher, Z. Tu, in Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (AISTATS, 2016), pp. 464–472.
18
M. A. Ranzato, C. Poultney, S. Chopra, Y. LeCun, in Advances in Neural Information Processing Systems 20 (NIPS 2007), J. C. Platt, D. Koller, Y. Singer, S. T. Roweis, Eds. (Curran Associates, 2007), pp. 1137–1144.
19
H. Xiao, K. Rasul, R. Vollgraf, Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv:1708.07747 [cs.LG] (25 August 2017).
20
S. Zeng, B. Zhang, J. Gou, Deep Collaborative Weight-based Classification. arXiv:1802.07589 [cs.CV] (21 February 2018).
21
22
Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, M. Soljačić, Deep learning with coherent nanophotonic circuits. Nat. Photonics 11, 441–446 (2017).
23
D. Psaltis, D. Brady, X.-G. Gu, S. Lin, Holography in artificial neural networks. Nature 343, 325–330 (1990).
24
K. H. Wagner, in Proceedings of Frontiers in Optics 2017 (OSA, 2017), FW2C.1.
25
B. J. Shastri et al., Principles of neuromorphic photonics. arXiv:1801.00016 [cs.ET] (29 December 2017).
26
M. Hermans, M. Burm, T. Van Vaerenbergh, J. Dambre, P. Bienstman, Trainable hardware for dynamical computing using error backpropagation through physical media. Nat. Commun. 6, 6729 (2015).
27
D. Brunner, M. C. Soriano, C. R. Mirasso, I. Fischer, Parallel photonic information processing at gigabyte per second data rates using transient states. Nat. Commun. 4, 1364 (2013).
28
A. Greenbaum, W. Luo, T.-W. Su, Z. Göröcs, L. Xue, S. O. Isikman, A. F. Coskun, O. Mudanyali, A. Ozcan, Imaging without lenses: Achievements and remaining challenges of wide-field on-chip microscopy. Nat. Methods 9, 889–895 (2012).
29
A. Ozcan, E. McLeod, Lensless imaging and sensing. Annu. Rev. Biomed. Eng. 18, 77–102 (2016).
30
M. Emons, K. Obata, T. Binhammer, A. Ovsianikov, B. N. Chichkov, U. Morgner, Two-photon polymerization technique with sub-50 nm resolution by sub-10 fs laser pulses. Opt. Mater. Express 2, 942–947 (2012).
31
D. P. Kingma, J. Ba, Adam: A method for stochastic optimization. arXiv:1412.6980 [cs.LG] (22 December 2014).
32
M. Kazhdan, H. Hoppe, Screened poisson surface reconstruction. ACM Trans. Graph. 32, 1–13 (2013).
33
34
P. Cignoni, M. Callieri, M. Corsini, M. Dellepiane, F. Ganovelli, G. Ranzuglia, “MeshLab: An open-source mesh processing tool” in Sixth Eurographics Italian Chapter Conference (2008), pp. 129–136.
35
J. W. Goodman, Introduction to Fourier Optics (Roberts and Company, 2005).
36
C. Trabelsi et al., Deep complex networks. arXiv:1705.09792 [cs.NE] (27 May 2017).
37
Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).
38
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014).
39
V. Bianchi, T. Carey, L. Viti, L. Li, E. H. Linfield, A. G. Davies, A. Tredicucci, D. Yoon, P. G. Karagiannidis, L. Lombardi, F. Tomarchio, A. C. Ferrari, F. Torrisi, M. S. Vitiello, Terahertz saturable absorbers from liquid phase exfoliation of graphite. Nat. Commun. 8, 15763 (2017).
40
A. Marini, J. D. Cox, F. J. García de Abajo, Theory of graphene saturable absorption. Phys. Rev. B 95, 125408 (2017).
41
X. Yin, T. Feng, Z. Liang, J. Li, Artificial Kerr-type medium using metamaterials. Opt. Express 20, 8543–8550 (2012).
42
Y. Xiao, H. Qian, Z. Liu, Nonlinear metasurface based on giant optical kerr response of gold quantum wells. ACS Photonics 5, 1654–1659 (2018).
43
N. Yu, F. Capasso, Flat optics with designer metasurfaces. Nat. Mater. 13, 139–150 (2014).
44
M. Khorasaninejad, W. T. Chen, R. C. Devlin, J. Oh, A. Y. Zhu, F. Capasso, Metalenses at visible wavelengths: Diffraction-limited focusing and subwavelength resolution imaging. Science 352, 1190–1194 (2016).
45
A. V. Kildishev, A. Boltasseva, V. M. Shalaev, Planar photonics with metasurfaces. Science 339, 1232009 (2013).
46
D. Grischkowsky, S. Keiding, M. Van Exter, C. Fattinger, Far-infrared time-domain spectroscopy with terahertz beams of dielectrics and semiconductors. J. Opt. Soc. Am. B 7, 2006–2015 (1990).

Information & Authors

Information

Published In

Science
Volume 361 | Issue 6406
7 September 2018

Copyright

Copyright © 2018 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works.
This is an article distributed under the terms of the Science Journals Default License.

Article versions

You are viewing the most recent version of this article.

Submission history

Received: 6 April 2018
Accepted: 12 July 2018
Published in print: 7 September 2018

Permissions

Request permissions for this article.

Acknowledgments

We thank D. Mengu, Z. Wei, X. Wang, and Y. Xiao of UCLA for assistance with coding. Funding: The Ozcan Group at UCLA acknowledges the support of the National Science Foundation and the Howard Hughes Medical Institute. Author contributions: A.O., X.L., and Y.R. conceived of the research; X.L., N.T.Y., Y.L., Y.R., M.V., and M.J. contributed to the experiments; X.L., N.T.Y., M.V., and Y.R. processed the data; A.O., X.L., M.V., N.T.Y., Y.R., Y.L., and M.J. prepared the manuscript; and A.O. initiated and supervised the research. Competing interests: A.O., X.L., and Y.R. are inventors of a patent application on D2NNs. Data and materials availability: All data and methods are present in the main text and supplementary materials.

Authors

Affiliations

Xing Lin* https://orcid.org/0000-0002-5560-7286
Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90095, USA.
Department of Bioengineering, University of California, Los Angeles, CA 90095, USA.
California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA.
View all articles by this author
Yair Rivenson*
Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90095, USA.
Department of Bioengineering, University of California, Los Angeles, CA 90095, USA.
California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA.
View all articles by this author
Nezih T. Yardimci https://orcid.org/0000-0002-3623-5835
Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90095, USA.
California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA.
View all articles by this author
Muhammed Veli
Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90095, USA.
Department of Bioengineering, University of California, Los Angeles, CA 90095, USA.
California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA.
View all articles by this author
Yi Luo
Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90095, USA.
Department of Bioengineering, University of California, Los Angeles, CA 90095, USA.
California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA.
View all articles by this author
Mona Jarrahi
Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90095, USA.
California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA.
View all articles by this author
Aydogan Ozcan† https://orcid.org/0000-0002-0717-683X [email protected]
Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90095, USA.
Department of Bioengineering, University of California, Los Angeles, CA 90095, USA.
California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA.
Department of Surgery, David Geffen School of Medicine, University of California, Los Angeles, CA 90095, USA.
View all articles by this author

Funding Information

Notes

*
These authors contributed equally to this work.
†Corresponding author. Email: [email protected]

Metrics & Citations

Metrics

Article Usage
Altmetrics

Citations

Export citation

Select the format you want to export the citation of this publication.

Cited by
    • Zhichao Sun,
    • Bijun Xu,
    • Fan Jin,
    • Gongxue Zhou,
    • Lu Lin,
    Machine Learning Approach for On-Demand Rapid Constructing Metasurface, IEEE Journal of Selected Topics in Quantum Electronics, 28, 4, (1-9), (2022).https://doi.org/10.1109/JSTQE.2021.3083565
    Crossref
    • Peipei Wang,
    • Wenjie Xiong,
    • Zebin Huang,
    • Yanliang He,
    • Junmin Liu,
    • Huapeng Ye,
    • Jiangnan Xiao,
    • Ying Li,
    • Dianyuan Fan,
    • Shuqing Chen,
    Diffractive Deep Neural Network for Optical Orbital Angular Momentum Multiplexing and Demultiplexing, IEEE Journal of Selected Topics in Quantum Electronics, 28, 4, (1-11), (2022).https://doi.org/10.1109/JSTQE.2021.3077907
    Crossref
    • Changming Wu,
    • Xiaoxuan Yang,
    • Heshan Yu,
    • Ruoming Peng,
    • Ichiro Takeuchi,
    • Yiran Chen,
    • Mo Li,
    Harnessing optoelectronic noises in a photonic generative network, Science Advances, 8, 3, (2022)./doi/10.1126/sciadv.abm2956
    Abstract
    • Philip Georgi,
    • Qunshuo Wei,
    • Basudeb Sain,
    • Christian Schlickriede,
    • Yongtian Wang,
    • Lingling Huang,
    • Thomas Zentgraf,
    Optical secret sharing with cascaded metasurface holography, Science Advances, 7, 16, (2021)./doi/10.1126/sciadv.abf9718
    Abstract
    • Peter R. Wiecha,
    • Arnaud Arbouet,
    • Christian Girard,
    • Otto L. Muskens,
    Deep learning in nano-photonics: inverse design and beyond, Photonics Research, 9, 5, (B182), (2021).https://doi.org/10.1364/PRJ.415960
    Crossref
    • Narayan Bhusal,
    • Sanjaya Lohani,
    • Chenglong You,
    • Mingyuan Hong,
    • Joshua Fabre,
    • Pengcheng Zhao,
    • Erin M. Knutson,
    • Ryan T. Glasser,
    • Omar S. Magaña‐Loaiza,
    Spatial Mode Correction of Single Photons Using Machine Learning, Advanced Quantum Technologies, 4, 3, (2000103), (2021).https://doi.org/10.1002/qute.202000103
    Crossref
    • Mengxi Tan,
    • Xingyuan Xu,
    • Bill Corcoran,
    • Jiayang Wu,
    • Andreas Boes,
    • thach Nguyen,
    • Sai T. Chu,
    • Brent Little,
    • Roberto Morandotti,
    • Arnan Mitchell,
    • David J. Moss,
    undefined, Integrated Optics: Design, Devices, Systems and Applications VI, (1), (2021).https://doi.org/10.1117/12.2588733
    Crossref
    • Chonghuai Ma,
    • Floris Laporte,
    • Joni Dambre,
    • Peter Bienstman,
    Addressing limited weight resolution in a fully optical neuromorphic reservoir computing readout, Scientific Reports, 11, 1, (2021).https://doi.org/10.1038/s41598-021-82720-4
    Crossref
    • Siddharth Buddhiraju,
    • Avik Dutt,
    • Momchil Minkov,
    • Ian A. D. Williamson,
    • Shanhui Fan,
    Arbitrary linear transformations for photons in the frequency synthetic dimension, Nature Communications, 12, 1, (2021).https://doi.org/10.1038/s41467-021-22670-7
    Crossref
    • Nikolaos Passalis,
    • George Mourgias-Alexandris,
    • Apostolos Tsakyridis,
    • Nikos Pleros,
    • Anastasios Tefas,
    Training Deep Photonic Convolutional Neural Networks With Sinusoidal Activations, IEEE Transactions on Emerging Topics in Computational Intelligence, 5, 3, (384-393), (2021).https://doi.org/10.1109/TETCI.2019.2923001
    Crossref
  11. See more

View Options

Get Access

Log in to view the full text

AAAS ID LOGIN

AAAS login provides access to Science for AAAS Members, and access to other journals in the Science family to users who have purchased individual subscriptions.

Log in via OpenAthens.
via OpenAthens
Log in via Shibboleth.
via Shibboleth
More options

Register for free to read this article

As a service to the community, this article is available for free. Login or register for free to read this article.

Purchase this issue in print

Buy a single issue of Science for just $15 USD.

View options

PDF format

Download this article as a PDF file

Download PDF

Media

Figures

Multimedia

Tables

Share

Share

Share article link

Share on social media

Current Issue

Go to Science
Table of Contents

LATEST NEWS