Hoodline and a former co-editor of TechCrunch.”,”slug”:”eric-eldon-2″,”links”:”homepage”:”http://about.me/eric.eldon”,”facebook”:”http://www.facebook.com/EricEldon”,”twitter”:”http://twitter.com/eldon”,”linkedin”:”http://www.linkedin.com/in/ericeldon”,”crunchbase”:”https://www.crunchbase.com/person/eric-eldon”,”position”:”Contributor”,”cbDescription”:”

]]>Eric Eldon is a writer and editor who has helped build and lead a variety of online media publications.He\’s currently the managing editor of Extra Crunch, TechCrunch\’s subscription product, where he helps to produce analysis and information that startups can use to make better decisions.rnrn

Disclosures: Eric is an extremely small shareholder in Nextdoor, via an eventual acquisition of a local news startup he had cofounded called Hoodline. His wife owns a very slightly larger number of shares in education company Udacity, and in Common Networks, and she is currently working at Google, focusing on developer education. He typically avoids covering these companies and topics, and discloses whenever relevant. He also owns a small amount of bitcoin for testing purposes.”,”cbAvatar”:”https://techcrunch.com/wp-content/uploads/2021/01/d21b3822c86c484fcbfc465fe5986f77.png.png”,”twitter”:”eldon”}],”author”:[“id”:786933,”name”:”Eric Eldon”,”url”:””,”description”:””,”link”:”https://techcrunch.com/author/eric-eldon/”,”slug”:”eric-eldon”,”avatar_urls”:”24″:”https://secure.gravatar.com/avatar/cba9774f944490f50a6ca7115cd043f9?s=24&d=identicon&r=g”,”48″:”https://secure.gravatar.com/avatar/cba9774f944490f50a6ca7115cd043f9?s=48&d=identicon&r=g”,”96″:”https://secure.gravatar.com/avatar/cba9774f944490f50a6ca7115cd043f9?s=96&d=identicon&r=g”,”links”:”homepage”:”http://about.me/eric.eldon”,”facebook”:”http://www.facebook.com/EricEldon”,”twitter”:”https://twitter.com/eldon”,”linkedin”:”http://www.linkedin.com/in/ericeldon”,”crunchbase”:”https://www.crunchbase.com/person/eric-eldon”,”position”:”Managing Editor, Extra Crunch”,”cbDescription”:”

Eric Eldon is a writer and editor who has helped build and lead a variety of online media publications.He\’s currently the managing editor of Extra Crunch, TechCrunch\’s subscription product, where he helps to produce analysis and information that startups can use to make better decisions.rnrn

Disclosures: Eric is an extremely small shareholder in Nextdoor, via an eventual acquisition of a local news startup he had cofounded called Hoodline. His wife owns a very slightly larger number of shares in education company Udacity, and in Common Networks, and she is currently working at Google, focusing on developer education. He typically avoids covering these companies and topics, and discloses whenever relevant. He also owns a small amount of bitcoin for testing purposes.”,”cbAvatar”:”https://techcrunch.com/wp-content/uploads/2021/01/d21b3822c86c484fcbfc465fe5986f77.png.png”,”twitter”:”eldon”,”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/tc/v1/users/786933″],”collection”:[“href”:”https://techcrunch.com/wp-json/tc/v1/users”]],”wp:featuredmedia”:[“id”:2167106,”date”:”2021-06-16T11:14:51″,”slug”:”top-view-of-businesspeople-talking-standing-on-painted-world-map-on-asphalt”,”type”:”attachment”,”link”:”https://techcrunch.com/2021/06/16/to-win-post-pandemic-startups-need-remote-first-growth-teams/top-view-of-businesspeople-talking-standing-on-painted-world-map-on-asphalt/”,”title”:”rendered”:”Top view of businesspeople talking. Standing on painted world map on asphalt”,”author”:133574325,”license”:”source_key”:”getty images”,”person”:”Klaus Vedfelt”,”person_url”:”https://www.gettyimages.com/search/photographer?family=creative&photographer=Klaus+Vedfelt”,”authors”:[133574325],”caption”:”rendered”:””,”alt_text”:”Group of young adults, photographed from above, on various painted tarmac surface, at sunrise.”,”media_type”:”image”,”mime_type”:”image/jpeg”,”media_details”:”width”:1500,”height”:1000,”file”:”2021/06/GettyImages-912015398.jpg”,”sizes”:”thumbnail”:”file”:”GettyImages-912015398.jpg?resize=150,100″,”width”:150,”height”:100,”mime_type”:”image/jpeg”,”source_url”:”https://techcrunch.com/wp-content/uploads/2021/06/GettyImages-912015398.jpg?w=150″,”medium”:”file”:”GettyImages-912015398.jpg?resize=300,200″,”width”:300,”height”:200,”mime_type”:”image/jpeg”,”source_url”:”https://techcrunch.com/wp-content/uploads/2021/06/GettyImages-912015398.jpg?w=300″,”medium_large”:”file”:”GettyImages-912015398.jpg?resize=768,512″,”width”:768,”height”:512,”mime_type”:”image/jpeg”,”source_url”:”https://techcrunch.com/wp-content/uploads/2021/06/GettyImages-912015398.jpg?w=1024″,”large”:”file”:”GettyImages-912015398.jpg?resize=680,453″,”width”:680,”height”:453,”mime_type”:”image/jpeg”,”source_url”:”https://techcrunch.com/wp-content/uploads/2021/06/GettyImages-912015398.jpg?w=680″,”guest-author-32″:”file”:”GettyImages-912015398.jpg?resize=32,32″,”width”:32,”height”:32,”mime_type”:”image/jpeg”,”source_url”:”https://techcrunch.com/wp-content/uploads/2021/06/GettyImages-912015398.jpg?w=32&h=32&crop=1″,”guest-author-50″:”file”:”GettyImages-912015398.jpg?resize=50,50″,”width”:50,”height”:50,”mime_type”:”image/jpeg”,”source_url”:”https://techcrunch.com/wp-content/uploads/2021/06/GettyImages-912015398.jpg?w=50&h=50&crop=1″,”guest-author-64″:”file”:”GettyImages-912015398.jpg?resize=64,64″,”width”:64,”height”:64,”mime_type”:”image/jpeg”,”source_url”:”https://techcrunch.com/wp-content/uploads/2021/06/GettyImages-912015398.jpg?w=64&h=64&crop=1″,”guest-author-96″:”file”:”GettyImages-912015398.jpg?resize=96,96″,”width”:96,”height”:96,”mime_type”:”image/jpeg”,”source_url”:”https://techcrunch.com/wp-content/uploads/2021/06/GettyImages-912015398.jpg?w=96&h=96&crop=1″,”guest-author-128″:”file”:”GettyImages-912015398.jpg?resize=128,128″,”width”:128,”height”:128,”mime_type”:”image/jpeg”,”source_url”:”https://techcrunch.com/wp-content/uploads/2021/06/GettyImages-912015398.jpg?w=128&h=128&crop=1″,”concierge-thumb”:”file”:”GettyImages-912015398.jpg?resize=50,33″,”width”:50,”height”:33,”mime_type”:”image/jpeg”,”source_url”:”https://techcrunch.com/wp-content/uploads/2021/06/GettyImages-912015398.jpg?w=50″,”full”:”file”:”GettyImages-912015398.jpg”,”width”:1024,”height”:683,”mime_type”:”image/jpeg”,”source_url”:”https://techcrunch.com/wp-content/uploads/2021/06/GettyImages-912015398.jpg”,”image_meta”:”aperture”:”0″,”credit”:”Getty Images”,”camera”:””,”caption”:”Group of young adults, photographed from above, on various painted tarmac surface, at sunrise.”,”created_timestamp”:”1489481355″,”copyright”:””,”focal_length”:”0″,”iso”:”0″,”shutter_speed”:”0″,”title”:”Top view of businesspeople talking. Standing on painted world map on asphalt”,”orientation”:”0″,”keywords”:[],”filesize”:1173374,”source_url”:”https://techcrunch.com/wp-content/uploads/2021/06/GettyImages-912015398.jpg”,”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/wp/v2/media/2167106″],”collection”:[“href”:”https://techcrunch.com/wp-json/wp/v2/media”],”about”:[“href”:”https://techcrunch.com/wp-json/wp/v2/types/attachment”],”replies”:[“embeddable”:true,”href”:”https://techcrunch.com/wp-json/wp/v2/comments?post=2167106″],”author”:[“embeddable”:true,”href”:”https://techcrunch.com/wp-json/tc/v1/users/133574325″]],”wp:term”:[[“id”:11160,”link”:”https://techcrunch.com/ecommerce/”,”name”:”eCommerce”,”slug”:”ecommerce”,”taxonomy”:”category”,”parent”:0,”rapidData”:”pt”:””,”pct”:””,”submenu_categories”:[],”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/wp/v2/categories/11160″],”collection”:[“href”:”https://techcrunch.com/wp-json/wp/v2/categories”],”about”:[“href”:”https://techcrunch.com/wp-json/wp/v2/taxonomies/category”],”wp:post_type”:[“href”:”https://techcrunch.com/wp-json/wp/v2/posts?categories=11160″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc-media-gallery?categories=11160″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc_video?categories=11160″],”curies”:[“name”:”wp”,”href”:”https://api.w.org/rel”,”templated”:true],”id”:576698407,”link”:”https://techcrunch.com/verified-experts/growth-marketing/”,”name”:”Growth Marketing”,”slug”:”growth-marketing”,”taxonomy”:”category”,”parent”:576633018,”rapidData”:”pt”:””,”pct”:””,”submenu_categories”:[],”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/wp/v2/categories/576698407″],”collection”:[“href”:”https://techcrunch.com/wp-json/wp/v2/categories”],”about”:[“href”:”https://techcrunch.com/wp-json/wp/v2/taxonomies/category”],”up”:[“embeddable”:true,”href”:”https://techcrunch.com/wp-json/wp/v2/categories/576633018″],”wp:post_type”:[“href”:”https://techcrunch.com/wp-json/wp/v2/posts?categories=576698407″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc-media-gallery?categories=576698407″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc_video?categories=576698407″],”curies”:[“name”:”wp”,”href”:”https://api.w.org/rel”,”templated”:true],”id”:20429,”link”:”https://techcrunch.com/startups/”,”name”:”Startups”,”slug”:”startups”,”taxonomy”:”category”,”parent”:0,”rapidData”:”pt”:””,”pct”:””,”submenu_categories”:[],”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/wp/v2/categories/20429″],”collection”:[“href”:”https://techcrunch.com/wp-json/wp/v2/categories”],”about”:[“href”:”https://techcrunch.com/wp-json/wp/v2/taxonomies/category”],”wp:post_type”:[“href”:”https://techcrunch.com/wp-json/wp/v2/posts?categories=20429″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc-media-gallery?categories=20429″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc_video?categories=20429″],”curies”:[“name”:”wp”,”href”:”https://api.w.org/rel”,”templated”:true],”id”:17396,”link”:”https://techcrunch.com/tc/”,”name”:”TC”,”slug”:”tc”,”taxonomy”:”category”,”parent”:0,”rapidData”:”pt”:””,”pct”:””,”submenu_categories”:[],”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/wp/v2/categories/17396″],”collection”:[“href”:”https://techcrunch.com/wp-json/wp/v2/categories”],”about”:[“href”:”https://techcrunch.com/wp-json/wp/v2/taxonomies/category”],”wp:post_type”:[“href”:”https://techcrunch.com/wp-json/wp/v2/posts?categories=17396″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc-media-gallery?categories=17396″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc_video?categories=17396″],”curies”:[“name”:”wp”,”href”:”https://api.w.org/rel”,”templated”:true],”id”:576633018,”link”:”https://techcrunch.com/verified-experts/”,”name”:”Verified Experts”,”slug”:”verified-experts”,”taxonomy”:”category”,”parent”:0,”rapidData”:”pt”:””,”pct”:””,”submenu_categories”:[],”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/wp/v2/categories/576633018″],”collection”:[“href”:”https://techcrunch.com/wp-json/wp/v2/categories”],”about”:[“href”:”https://techcrunch.com/wp-json/wp/v2/taxonomies/category”],”wp:post_type”:[“href”:”https://techcrunch.com/wp-json/wp/v2/posts?categories=576633018″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc-media-gallery?categories=576633018″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc_video?categories=576633018″],”curies”:[“name”:”wp”,”href”:”https://api.w.org/rel”,”templated”:true]],[“id”:576747132,”link”:”https://techcrunch.com/tag/collaboration-tools/”,”name”:”collaboration tools”,”slug”:”collaboration-tools”,”taxonomy”:”post_tag”,”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags/576747132″],”collection”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags”],”about”:[“href”:”https://techcrunch.com/wp-json/wp/v2/taxonomies/post_tag”],”wp:post_type”:[“href”:”https://techcrunch.com/wp-json/wp/v2/posts?tags=576747132″,”href”:”https://techcrunch.com/wp-json/wp/v2/battlefield-companies?tags=576747132″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc-media-gallery?tags=576747132″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc_video?tags=576747132″],”curies”:[“name”:”wp”,”href”:”https://api.w.org/rel”,”templated”:true],”id”:154578,”link”:”https://techcrunch.com/tag/digital-marketing/”,”name”:”digital marketing”,”slug”:”digital-marketing”,”taxonomy”:”post_tag”,”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags/154578″],”collection”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags”],”about”:[“href”:”https://techcrunch.com/wp-json/wp/v2/taxonomies/post_tag”],”wp:post_type”:[“href”:”https://techcrunch.com/wp-json/wp/v2/posts?tags=154578″,”href”:”https://techcrunch.com/wp-json/wp/v2/battlefield-companies?tags=154578″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc-media-gallery?tags=154578″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc_video?tags=154578″],”curies”:[“name”:”wp”,”href”:”https://api.w.org/rel”,”templated”:true],”id”:9881,”link”:”https://techcrunch.com/tag/e-commerce/”,”name”:”e-commerce”,”slug”:”e-commerce”,”taxonomy”:”post_tag”,”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags/9881″],”collection”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags”],”about”:[“href”:”https://techcrunch.com/wp-json/wp/v2/taxonomies/post_tag”],”wp:post_type”:[“href”:”https://techcrunch.com/wp-json/wp/v2/posts?tags=9881″,”href”:”https://techcrunch.com/wp-json/wp/v2/battlefield-companies?tags=9881″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc-media-gallery?tags=9881″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc_video?tags=9881″],”curies”:[“name”:”wp”,”href”:”https://api.w.org/rel”,”templated”:true],”id”:183,”link”:”https://techcrunch.com/tag/entrepreneurship/”,”name”:”entrepreneurship”,”slug”:”entrepreneurship”,”taxonomy”:”post_tag”,”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags/183″],”collection”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags”],”about”:[“href”:”https://techcrunch.com/wp-json/wp/v2/taxonomies/post_tag”],”wp:post_type”:[“href”:”https://techcrunch.com/wp-json/wp/v2/posts?tags=183″,”href”:”https://techcrunch.com/wp-json/wp/v2/battlefield-companies?tags=183″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc-media-gallery?tags=183″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc_video?tags=183″],”curies”:[“name”:”wp”,”href”:”https://api.w.org/rel”,”templated”:true],”id”:99242442,”link”:”https://techcrunch.com/tag/growth-hacking/”,”name”:”growth hacking”,”slug”:”growth-hacking”,”taxonomy”:”post_tag”,”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags/99242442″],”collection”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags”],”about”:[“href”:”https://techcrunch.com/wp-json/wp/v2/taxonomies/post_tag”],”wp:post_type”:[“href”:”https://techcrunch.com/wp-json/wp/v2/posts?tags=99242442″,”href”:”https://techcrunch.com/wp-json/wp/v2/battlefield-companies?tags=99242442″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc-media-gallery?tags=99242442″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc_video?tags=99242442″],”curies”:[“name”:”wp”,”href”:”https://api.w.org/rel”,”templated”:true],”id”:576639396,”link”:”https://techcrunch.com/tag/growth-marketing/”,”name”:”growth marketing”,”slug”:”growth-marketing”,”taxonomy”:”post_tag”,”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags/576639396″],”collection”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags”],”about”:[“href”:”https://techcrunch.com/wp-json/wp/v2/taxonomies/post_tag”],”wp:post_type”:[“href”:”https://techcrunch.com/wp-json/wp/v2/posts?tags=576639396″,”href”:”https://techcrunch.com/wp-json/wp/v2/battlefield-companies?tags=576639396″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc-media-gallery?tags=576639396″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc_video?tags=576639396″],”curies”:[“name”:”wp”,”href”:”https://api.w.org/rel”,”templated”:true],”id”:576735516,”link”:”https://techcrunch.com/tag/susan-su/”,”name”:”susan su”,”slug”:”susan-su”,”taxonomy”:”post_tag”,”_links”:”self”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags/576735516″],”collection”:[“href”:”https://techcrunch.com/wp-json/wp/v2/tags”],”about”:[“href”:”https://techcrunch.com/wp-json/wp/v2/taxonomies/post_tag”],”wp:post_type”:[“href”:”https://techcrunch.com/wp-json/wp/v2/posts?tags=576735516″,”href”:”https://techcrunch.com/wp-json/wp/v2/battlefield-companies?tags=576735516″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc-media-gallery?tags=576735516″,”href”:”https://techcrunch.com/wp-json/wp/v2/tc_video?tags=576735516″],”curies”:[“name”:”wp”,”href”:”https://api.w.org/rel”,”templated”:true]],[],[],[],[]]}}]},”isAdminBar”:””,”marketoEventId”:”1004″,”marketoBrandStudioId”:”1740″,”menus”:”main”:[“id”:2116132,”order”:1,”parent”:0,”text”:”TC Sessions: Mobility 2021″,”url”:”https://techcrunch.com/events/tc-sessions-mobility-2021?promo=leftrail&display=true”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:2116132,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:335981,”order”:2,”parent”:0,”text”:”Startups”,”url”:”https://techcrunch.com/startups/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:20429,”object”:”category”,”type”:”taxonomy”,”type_label”:”Category”,”children”:[],”id”:335986,”order”:3,”parent”:0,”text”:”Videos”,”url”:”/video/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:335986,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1761192,”order”:4,”parent”:0,”text”:”Audio”,”url”:”https://techcrunch.com/pages/podcasts/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1760429,”object”:”page”,”type”:”post_type”,”type_label”:”Page”,”children”:[],”id”:1599992,”order”:5,”parent”:0,”text”:”Newsletters”,”url”:”/newsletters”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1599992,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1781882,”order”:6,”parent”:0,”text”:”Extra Crunch”,”url”:”/extracrunch/?tpcc=ecleftnav”,”attr”:””,”target”:””,”classes”:[“menu__item–extra-crunch”],”description”:””,”object_id”:1781882,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1599991,”order”:7,”parent”:0,”text”:”EC-1s”,”url”:”/ec1/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1599991,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:2016552,”order”:8,”parent”:0,”text”:”The TC List”,”url”:”/the-techcrunch-list/?tpcc=tclistleftnav”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:2016552,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1752899,”order”:9,”parent”:0,”text”:”Advertise”,”url”:”https://techcrunch.com/pages/advertisement-events-calendar/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1416499,”object”:”page”,”type”:”post_type”,”type_label”:”Page”,”children”:[],”id”:1599990,”order”:10,”parent”:0,”text”:”Events”,”url”:”/events/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1599990,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[]],”startup-battlefield”:[“id”:1602061,”order”:1,”parent”:0,”text”:”About”,”url”:”/startup-battlefield/about/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1602061,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1602062,”order”:2,”parent”:0,”text”:”FAQ”,”url”:”/startup-battlefield/faq/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1602062,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1602065,”order”:3,”parent”:0,”text”:”Battlefield Home”,”url”:”/startup-battlefield/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1602065,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1602063,”order”:4,”parent”:0,”text”:”Leaderboard”,”url”:”/startup-battlefield/leaderboard/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1602063,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1787197,”order”:5,”parent”:0,”text”:”Apply Now”,”url”:”https://apply.techcrunch.com”,”attr”:””,”target”:””,”classes”:[“apply-now-link”],”description”:””,”object_id”:1787197,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[]],”footer”:[“id”:1996921,”order”:1,”parent”:0,”text”:”About”,”url”:”#”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1996921,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[“id”:1996936,”order”:2,”parent”:1996921,”text”:”TechCrunch”,”url”:”https://techcrunch.com”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1996936,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1996935,”order”:3,”parent”:1996921,”text”:”Staff”,”url”:”/pages/about-techcrunch”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1996935,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1996938,”order”:4,”parent”:1996921,”text”:”Contact Us”,”url”:”https://techcrunch.com/pages/contact-us/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:672,”object”:”page”,”type”:”post_type”,”type_label”:”Page”,”children”:[],”id”:1996933,”order”:5,”parent”:1996921,”text”:”Advertise”,”url”:”https://techcrunch.com/pages/advertisement-events-calendar/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1416499,”object”:”page”,”type”:”post_type”,”type_label”:”Page”,”children”:[]],”id”:1996922,”order”:6,”parent”:0,”text”:”Legal”,”url”:”#”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1996922,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[“id”:1996932,”order”:7,”parent”:1996922,”text”:”Privacy Policy”,”url”:”https://www.verizonmedia.com/policies/us/en/verizonmedia/privacy/index.html”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1996932,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1996931,”order”:8,”parent”:1996922,”text”:”Terms of Service”,”url”:”https://www.verizonmedia.com/policies/us/en/verizonmedia/terms/otos/index.html”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1996931,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:2003673,”order”:9,”parent”:1996922,”text”:”Extra Crunch Terms”,”url”:”https://techcrunch.com/pages/extra-crunch-terms-of-service/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:2000290,”object”:”page”,”type”:”post_type”,”type_label”:”Page”,”children”:[],”id”:1996930,”order”:10,”parent”:1996922,”text”:”Privacy Rights”,”url”:”#”,”attr”:””,”target”:””,”classes”:[“do-not-sell-link”],”description”:””,”object_id”:1996930,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1996929,”order”:11,”parent”:1996922,”text”:”Code of Conduct”,”url”:”https://techcrunch.com/pages/code-of-conduct/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1678717,”object”:”page”,”type”:”post_type”,”type_label”:”Page”,”children”:[]],”id”:1996924,”order”:12,”parent”:0,”text”:”International”,”url”:”#”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1996924,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[“id”:1996925,”order”:13,”parent”:1996924,”text”:”Japan”,”url”:”http://jp.techcrunch.com/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1996925,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[]]],”submenu”:[“id”:1996941,”order”:1,”parent”:0,”text”:”Startup Battlefield”,”url”:”/startup-battlefield”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1996941,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1996942,”order”:2,”parent”:0,”text”:”Sponsored Content”,”url”:”/sponsored”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1996942,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1996944,”order”:3,”parent”:0,”text”:”Include”,”url”:”/include/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1996944,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1996945,”order”:4,”parent”:0,”text”:”Crunchbase”,”url”:”http://crunchbase.com/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1996945,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1996940,”order”:5,”parent”:0,”text”:”Crunchboard”,”url”:”http://crunchboard.com”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1996940,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:2009102,”order”:6,”parent”:0,”text”:”Contact Us”,”url”:”https://techcrunch.com/pages/contact-us/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:672,”object”:”page”,”type”:”post_type”,”type_label”:”Page”,”children”:[]],”include”:[“id”:2029903,”order”:1,”parent”:0,”text”:”Include Home”,”url”:”/include/”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:2029903,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1601974,”order”:2,”parent”:0,”text”:”Our Mission”,”url”:”/include/mission-statement”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1601974,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1601975,”order”:3,”parent”:0,”text”:”Office Hours”,”url”:”/include/office-hours”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1601975,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[]],”video-hub”:[“id”:1605165,”order”:1,”parent”:0,”text”:”News”,”url”:”/shows/tctv-news”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605165,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605166,”order”:2,”parent”:0,”text”:”Gadgets”,”url”:”/shows/gadgets”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605166,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605167,”order”:3,”parent”:0,”text”:”Features”,”url”:”/shows/features”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605167,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605168,”order”:4,”parent”:0,”text”:”Reviews”,”url”:”/shows/reviews”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605168,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605169,”order”:5,”parent”:0,”text”:”Interviews”,”url”:”/shows/interviews”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605169,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605170,”order”:6,”parent”:0,”text”:”Apps”,”url”:”/shows/apps”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605170,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605171,”order”:7,”parent”:0,”text”:”Disrupt”,”url”:”/shows/techcrunch-disrupt”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605171,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605172,”order”:8,”parent”:0,”text”:”Battlefield”,”url”:”/shows/techcrunch-battlefield”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605172,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605173,”order”:9,”parent”:0,”text”:”Sessions”,”url”:”/shows/tc-sessions”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605173,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605174,”order”:10,”parent”:0,”text”:”Crunch Report”,”url”:”/shows/crunch-report”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605174,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605175,”order”:11,”parent”:0,”text”:”Judah vs the Machines”,”url”:”/shows/judah-vs-the-machines”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605175,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605176,”order”:12,”parent”:0,”text”:”Down Round”,”url”:”/shows/down-round”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605176,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605177,”order”:13,”parent”:0,”text”:”Trust Disrupted”,”url”:”/shows/trust-disrupted”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605177,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605178,”order”:14,”parent”:0,”text”:”Built in Brooklyn”,”url”:”/shows/built-in-brooklyn”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605178,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605179,”order”:15,”parent”:0,”text”:”Inside Jobs”,”url”:”/shows/inside-jobs”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605179,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[],”id”:1605180,”order”:16,”parent”:0,”text”:”TC Cribs”,”url”:”/shows/tc-cribs”,”attr”:””,”target”:””,”classes”:[“”],”description”:””,”object_id”:1605180,”object”:”custom”,”type”:”custom”,”type_label”:”Custom Link”,”children”:[]],”entities”:{“posts”:[{“id”:2167273,”date”:”2021-06-16T16:22:23″,”date_gmt”:”2021-06-16T23:22:23″,”guid”:”rendered”:”https://techcrunch.com/?p=2167273″,”modified”:”2021-06-16T16:22:23″,”modified_gmt”:”2021-06-16T23:22:23″,”slug”:”a-look-inside-googles-first-store-opening-in-nycs-chelsea-neighborhood-tomorrow”,”status”:”publish”,”type”:”post”,”link”:”https://techcrunch.com/2021/06/16/a-look-inside-googles-first-store-opening-in-nycs-chelsea-neighborhood-tomorrow/”,”title”:”rendered”:”A look inside Googleu2019s first store, opening in NYCu2019s Chelsea neighborhood tomorrow”,”content”:{“rendered”:”

There have been plenty of pop-ups over the years, but tomorrow Googleu2019s first store opens in NYC’s Chelsea neighborhood. The brick and mortar model finds the company joining peers like Apple, Microsoft, Samsung and even Amazon, all of whom have a retail presence in Manhattan, including several just around the corner from Googleu2019s new digs.n

The new space, which opens tomorrow morning at 10 a.m. local time, fills 5,000 square feet of selling space in Googleu2019s big, pricey West Side real estate investment. The retail location was previously occupied by a Post Office and Starbucks, which vacated the premises once their leases expired under their new corporate landlord.n

Gradient descent is an optimization algorithm that follows the negative gradient of an objective function in order to locate the minimum of the function.

A limitation of gradient descent is that a single step size (learning rate) is used for all input variables. Extensions to gradient descent like the Adaptive Movement Estimation (Adam) algorithm use a separate step size for each input variable but may result in a step size that rapidly decreases to very small values.

**AMSGrad** is an extension to the Adam version of gradient descent that attempts to improve the convergence properties of the algorithm, avoiding large abrupt changes in the learning rate for each input variable.

In this tutorial, you will discover how to develop gradient descent optimization with AMSGrad from scratch.

After completing this tutorial, you will know:

- Gradient descent is an optimization algorithm that uses the gradient of the objective function to navigate the search space.
- AMSGrad is an extension of the Adam version of gradient descent designed to accelerate the optimization process.
- How to implement the AMSGrad optimization algorithm from scratch and apply it to an objective function and evaluate the results.

Let’s get started.

This tutorial is divided into three parts; they are:

- Gradient Descent
- AMSGrad Optimization Algorithm
- Gradient Descent With AMSGrad
- Two-Dimensional Test Problem
- Gradient Descent Optimization With AMSGrad
- Visualization of AMSGrad Optimization

Gradient descent is an optimization algorithm.

It is technically referred to as a first-order optimization algorithm as it explicitly makes use of the first-order derivative of the target objective function.

First-order methods rely on gradient information to help direct the search for a minimum …

— Page 69, Algorithms for Optimization, 2019.

The first-order derivative, or simply the “derivative,” is the rate of change or slope of the target function at a specific point, e.g. for a specific input.

If the target function takes multiple input variables, it is referred to as a multivariate function and the input variables can be thought of as a vector. In turn, the derivative of a multivariate target function may also be taken as a vector and is referred to generally as the gradient.

**Gradient**: First-order derivative for a multivariate objective function.

The derivative or the gradient points in the direction of the steepest ascent of the target function for a specific input.

Gradient descent refers to a minimization optimization algorithm that follows the negative of the gradient downhill of the target function to locate the minimum of the function.

The gradient descent algorithm requires a target function that is being optimized and the derivative function for the objective function. The target function *f()* returns a score for a given set of inputs, and the derivative function *f'()* gives the derivative of the target function for a given set of inputs.

The gradient descent algorithm requires a starting point (*x*) in the problem, such as a randomly selected point in the input space.

The derivative is then calculated and a step is taken in the input space that is expected to result in a downhill movement in the target function, assuming we are minimizing the target function.

A downhill movement is made by first calculating how far to move in the input space, calculated as the step size (called alpha or the learning rate) multiplied by the gradient. This is then subtracted from the current point, ensuring we move against the gradient, or down the target function.

- x(t) = x(t-1) – step_size * f'(x(t))

The steeper the objective function at a given point, the larger the magnitude of the gradient, and in turn, the larger the step taken in the search space. The size of the step taken is scaled using a step size hyperparameter.

**Step Size**: Hyperparameter that controls how far to move in the search space against the gradient each iteration of the algorithm.

If the step size is too small, the movement in the search space will be small and the search will take a long time. If the step size is too large, the search may bounce around the search space and skip over the optima.

Now that we are familiar with the gradient descent optimization algorithm, let’s take a look at the AMSGrad algorithm.

AMSGrad algorithm is an extension to the Adaptive Movement Estimation (Adam) optimization algorithm. More broadly, is an extension to the Gradient Descent Optimization algorithm.

The algorithm was described in the 2018 paper by J. Sashank, et al. titled “On the Convergence of Adam and Beyond.”

Generally, Adam automatically adapts a separate step size (learning rate) for each parameter in the optimization problem.

A limitation of Adam is that it can both decrease the step size when getting close to the optima, which is good, but it also increases the step size in some cases, which is bad.

AdaGrad addresses this specifically.

… ADAM aggressively increases the learning rate, however, […] this can be detrimental to the overall performance of the algorithm. […] In contrast, AMSGRAD neither increases nor decreases the learning rate and furthermore, decreases vt which can potentially lead to non-decreasing learning rate even if gradient is large in the future iterations.

— On the Convergence of Adam and Beyond, 2018.

AdaGrad is an extension to Adam that maintains a maximum of the second moment vector and uses it to bias-correct the gradient used to update the parameter, instead of the moment vector itself. This helps to stop the optimization slowing down too quickly (e.g. premature convergence).

The key difference of AMSGRAD with ADAM is that it maintains the maximum of all vt until the present time step and uses this maximum value for normalizing the running average of the gradient instead of vt in ADAM.

— On the Convergence of Adam and Beyond, 2018.

Let’s step through each element of the algorithm.

First, we must maintain a first and second moment vector as well as a max second moment vector for each parameter being optimized as part of the search, referred to as *m* and *v* (Geek letter nu but we will use v), and *vhat* respectively.

They are initialized to 0.0 at the start of the search.

The algorithm is executed iteratively over time t starting at *t=1*, and each iteration involves calculating a new set of parameter values *x*, e.g. going from *x(t-1)* to *x(t)*.

It is perhaps easy to understand the algorithm if we focus on updating one parameter, which generalizes to updating all parameters via vector operations.

First, the gradients (partial derivatives) are calculated for the current time step.

Next, the first moment vector is updated using the gradient and a hyperparameter *beta1*.

- m(t) = beta1(t) * m(t-1) + (1 – beta1(t)) * g(t)

The *beta1* hyperparameter can be held constant or can be decayed exponentially over the course of the search, such as:

Or, alternately:

The second moment vector is updated using the square of the gradient and a hyperparameter *beta2*.

- v(t) = beta2 * v(t-1) + (1 – beta2) * g(t)^2

Next, the maximum for the second moment vector is updated.

- vhat(t) = max(vhat(t-1), v(t))

Where *max()* calculates the maximum of the provided values.

The parameter value can then be updated using the calculated terms and the step size hyperparameter *alpha*.

- x(t) = x(t-1) – alpha(t) * m(t) / sqrt(vhat(t)))

Where *sqrt()* is the square root function.

The step size may also be held constant or decayed exponentially.

To review, there are three hyperparameters for the algorithm; they are:

**alpha**: Initial step size (learning rate), a typical value is 0.002.**beta1**: Decay factor for first momentum, a typical value is 0.9.**beta2**: Decay factor for infinity norm, a typical value is 0.999.

And that’s it.

For full derivation of the AMSGrad algorithm in the context of the Adam algorithm, I recommend reading the paper.

Next, let’s look at how we might implement the algorithm from scratch in Python.

In this section, we will explore how to implement the gradient descent optimization algorithm with AMSGrad Momentum.

First, let’s define an optimization function.

We will use a simple two-dimensional function that squares the input of each dimension and define the range of valid inputs from -1.0 to 1.0.

The *objective()* function below implements this.

# objective function def objective(x, y): return x**2.0 + y**2.0 |

We can create a three-dimensional plot of the dataset to get a feeling for the curvature of the response surface.

The complete example of plotting the objective function is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# 3d plot of the test function from numpy import arange from numpy import meshgrid from matplotlib import pyplot
# objective function def objective(x, y): return x**2.0 + y**2.0
# define range for input r_min, r_max = –1.0, 1.0 # sample input range uniformly at 0.1 increments xaxis = arange(r_min, r_max, 0.1) yaxis = arange(r_min, r_max, 0.1) # create a mesh from the axis x, y = meshgrid(xaxis, yaxis) # compute targets results = objective(x, y) # create a surface plot with the jet color scheme figure = pyplot.figure() axis = figure.gca(projection=‘3d’) axis.plot_surface(x, y, results, cmap=‘jet’) # show the plot pyplot.show() |

Running the example creates a three-dimensional surface plot of the objective function.

We can see the familiar bowl shape with the global minima at f(0, 0) = 0.

We can also create a two-dimensional plot of the function. This will be helpful later when we want to plot the progress of the search.

The example below creates a contour plot of the objective function.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# contour plot of the test function from numpy import asarray from numpy import arange from numpy import meshgrid from matplotlib import pyplot
# objective function def objective(x, y): return x**2.0 + y**2.0
# define range for input bounds = asarray([[–1.0, 1.0], [–1.0, 1.0]]) # sample input range uniformly at 0.1 increments xaxis = arange(bounds[0,0], bounds[0,1], 0.1) yaxis = arange(bounds[1,0], bounds[1,1], 0.1) # create a mesh from the axis x, y = meshgrid(xaxis, yaxis) # compute targets results = objective(x, y) # create a filled contour plot with 50 levels and jet color scheme pyplot.contourf(x, y, results, levels=50, cmap=‘jet’) # show the plot pyplot.show() |

Running the example creates a two-dimensional contour plot of the objective function.

We can see the bowl shape compressed to contours shown with a color gradient. We will use this plot to plot the specific points explored during the progress of the search.

Now that we have a test objective function, let’s look at how we might implement the AMSGrad optimization algorithm.

We can apply gradient descent with AMSGrad to the test problem.

First, we need a function that calculates the derivative for this function.

The derivative of *x^2* is *x * 2* in each dimension.

The *derivative()* function implements this below.

# derivative of objective function def derivative(x, y): return asarray([x * 2.0, y * 2.0]) |

Next, we can implement gradient descent optimization with AMSGrad.

First, we can select a random point in the bounds of the problem as a starting point for the search.

This assumes we have an array that defines the bounds of the search with one row for each dimension and the first column defines the minimum and the second column defines the maximum of the dimension.

... # generate an initial point x = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] – bounds[:, 0]) |

Next, we need to initialize the moment vectors.

... # initialize moment vectors m = [0.0 for _ in range(bounds.shape[0])] v = [0.0 for _ in range(bounds.shape[0])] vhat = [0.0 for _ in range(bounds.shape[0])] |

We then run a fixed number of iterations of the algorithm defined by the “*n_iter*” hyperparameter.

... # run iterations of gradient descent for t in range(n_iter): ... |

The first step is to calculate the derivative for the current set of parameters.

... # calculate gradient g(t) g = derivative(x[0], x[1]) |

Next, we need to perform the AMSGrad update calculations. We will perform these calculations one variable at a time using an imperative programming style for readability.

In practice, I recommend using NumPy vector operations for efficiency.

... # build a solution one variable at a time for i in range(x.shape[0]): ... |

First, we need to calculate the first moment vector.

... # m(t) = beta1(t) * m(t-1) + (1 – beta1(t)) * g(t) m[i] = beta1**(t+1) * m[i] + (1.0 – beta1**(t+1)) * g[i] |

Next, we need to calculate the second moment vector.

... # v(t) = beta2 * v(t-1) + (1 – beta2) * g(t)^2 v[i] = (beta2 * v[i]) + (1.0 – beta2) * g[i]**2 |

Then the maximum of the second moment vector with the previous iteration and the current value.

... # vhat(t) = max(vhat(t-1), v(t)) vhat[i] = max(vhat[i], v[i]) |

Finally, we can calculate the new value for the variable.

... # x(t) = x(t-1) – alpha(t) * m(t) / sqrt(vhat(t))) x[i] = x[i] – alpha * m[i] / sqrt(vhat[i]) |

We may want to add a small value to the denominator to avoid a divide by zero error; for example:

... # x(t) = x(t-1) – alpha(t) * m(t) / sqrt(vhat(t))) x[i] = x[i] – alpha * m[i] / (sqrt(vhat[i]) + 1e–8) |

This is then repeated for each parameter that is being optimized.

At the end of the iteration, we can evaluate the new parameter values and report the performance of the search.

... # evaluate candidate point score = objective(x[0], x[1]) # report progress print(‘>%d f(%s) = %.5f’ % (t, x, score)) |

We can tie all of this together into a function named *amsgrad()* that takes the names of the objective and derivative functions as well as the algorithm hyperparameters, and returns the best solution found at the end of the search and its evaluation.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
# gradient descent algorithm with amsgrad def amsgrad(objective, derivative, bounds, n_iter, alpha, beta1, beta2): # generate an initial point x = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] – bounds[:, 0]) # initialize moment vectors m = [0.0 for _ in range(bounds.shape[0])] v = [0.0 for _ in range(bounds.shape[0])] vhat = [0.0 for _ in range(bounds.shape[0])] # run the gradient descent for t in range(n_iter): # calculate gradient g(t) g = derivative(x[0], x[1]) # update variables one at a time for i in range(x.shape[0]): # m(t) = beta1(t) * m(t-1) + (1 – beta1(t)) * g(t) m[i] = beta1**(t+1) * m[i] + (1.0 – beta1**(t+1)) * g[i] # v(t) = beta2 * v(t-1) + (1 – beta2) * g(t)^2 v[i] = (beta2 * v[i]) + (1.0 – beta2) * g[i]**2 # vhat(t) = max(vhat(t-1), v(t)) vhat[i] = max(vhat[i], v[i]) # x(t) = x(t-1) – alpha(t) * m(t) / sqrt(vhat(t))) x[i] = x[i] – alpha * m[i] / (sqrt(vhat[i]) + 1e–8) # evaluate candidate point score = objective(x[0], x[1]) # report progress print(‘>%d f(%s) = %.5f’ % (t, x, score)) return [x, score] |

We can then define the bounds of the function and the hyperparameters and call the function to perform the optimization.

In this case, we will run the algorithm for 100 iterations with an initial learning rate of 0.007, beta of 0.9, and a beta2 of 0.99, found after a little trial and error.

... # seed the pseudo random number generator seed(1) # define range for input bounds = asarray([[–1.0, 1.0], [–1.0, 1.0]]) # define the total iterations n_iter = 100 # steps size alpha = 0.007 # factor for average gradient beta1 = 0.9 # factor for average squared gradient beta2 = 0.99 # perform the gradient descent search with amsgrad best, score = amsgrad(objective, derivative, bounds, n_iter, alpha, beta1, beta2) |

At the end of the run, we will report the best solution found.

... # summarize the result print(‘Done!’) print(‘f(%s) = %f’ % (best, score)) |

Tying all of this together, the complete example of AMSGrad gradient descent applied to our test problem is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 |
# gradient descent optimization with amsgrad for a two-dimensional test function from math import sqrt from numpy import asarray from numpy.random import rand from numpy.random import seed
# objective function def objective(x, y): return x**2.0 + y**2.0
# derivative of objective function def derivative(x, y): return asarray([x * 2.0, y * 2.0])
# gradient descent algorithm with amsgrad def amsgrad(objective, derivative, bounds, n_iter, alpha, beta1, beta2): # generate an initial point x = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] – bounds[:, 0]) # initialize moment vectors m = [0.0 for _ in range(bounds.shape[0])] v = [0.0 for _ in range(bounds.shape[0])] vhat = [0.0 for _ in range(bounds.shape[0])] # run the gradient descent for t in range(n_iter): # calculate gradient g(t) g = derivative(x[0], x[1]) # update variables one at a time for i in range(x.shape[0]): # m(t) = beta1(t) * m(t-1) + (1 – beta1(t)) * g(t) m[i] = beta1**(t+1) * m[i] + (1.0 – beta1**(t+1)) * g[i] # v(t) = beta2 * v(t-1) + (1 – beta2) * g(t)^2 v[i] = (beta2 * v[i]) + (1.0 – beta2) * g[i]**2 # vhat(t) = max(vhat(t-1), v(t)) vhat[i] = max(vhat[i], v[i]) # x(t) = x(t-1) – alpha(t) * m(t) / sqrt(vhat(t))) x[i] = x[i] – alpha * m[i] / (sqrt(vhat[i]) + 1e–8) # evaluate candidate point score = objective(x[0], x[1]) # report progress print(‘>%d f(%s) = %.5f’ % (t, x, score)) return [x, score]
# seed the pseudo random number generator seed(1) # define range for input bounds = asarray([[–1.0, 1.0], [–1.0, 1.0]]) # define the total iterations n_iter = 100 # steps size alpha = 0.007 # factor for average gradient beta1 = 0.9 # factor for average squared gradient beta2 = 0.99 # perform the gradient descent search with amsgrad best, score = amsgrad(objective, derivative, bounds, n_iter, alpha, beta1, beta2) print(‘Done!’) print(‘f(%s) = %f’ % (best, score)) |

Running the example applies the optimization algorithm with AMSGrad to our test problem and reports the performance of the search for each iteration of the algorithm.

**Note**: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, we can see that a near-optimal solution was found after perhaps 90 iterations of the search, with input values near 0.0 and 0.0, evaluating to 0.0.

… >90 f([-5.74880707e-11 2.16227707e-03]) = 0.00000 >91 f([-4.53359947e-11 2.03974264e-03]) = 0.00000 >92 f([-3.57526928e-11 1.92415218e-03]) = 0.00000 >93 f([-2.81951584e-11 1.81511216e-03]) = 0.00000 >94 f([-2.22351711e-11 1.71225138e-03]) = 0.00000 >95 f([-1.75350316e-11 1.61521966e-03]) = 0.00000 >96 f([-1.38284262e-11 1.52368665e-03]) = 0.00000 >97 f([-1.09053366e-11 1.43734076e-03]) = 0.00000 >98 f([-8.60013947e-12 1.35588802e-03]) = 0.00000 >99 f([-6.78222208e-12 1.27905115e-03]) = 0.00000 Done! f([-6.78222208e-12 1.27905115e-03]) = 0.000002 |

We can plot the progress of the AMSGrad search on a contour plot of the domain.

This can provide an intuition for the progress of the search over the iterations of the algorithm.

We must update the *amsgrad()* function to maintain a list of all solutions found during the search, then return this list at the end of the search.

The updated version of the function with these changes is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# gradient descent algorithm with amsgrad def amsgrad(objective, derivative, bounds, n_iter, alpha, beta1, beta2): solutions = list() # generate an initial point x = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] – bounds[:, 0]) # initialize moment vectors m = [0.0 for _ in range(bounds.shape[0])] v = [0.0 for _ in range(bounds.shape[0])] vhat = [0.0 for _ in range(bounds.shape[0])] # run the gradient descent for t in range(n_iter): # calculate gradient g(t) g = derivative(x[0], x[1]) # update variables one at a time for i in range(x.shape[0]): # m(t) = beta1(t) * m(t-1) + (1 – beta1(t)) * g(t) m[i] = beta1**(t+1) * m[i] + (1.0 – beta1**(t+1)) * g[i] # v(t) = beta2 * v(t-1) + (1 – beta2) * g(t)^2 v[i] = (beta2 * v[i]) + (1.0 – beta2) * g[i]**2 # vhat(t) = max(vhat(t-1), v(t)) vhat[i] = max(vhat[i], v[i]) # x(t) = x(t-1) – alpha(t) * m(t) / sqrt(vhat(t))) x[i] = x[i] – alpha * m[i] / (sqrt(vhat[i]) + 1e–8) # evaluate candidate point score = objective(x[0], x[1]) # keep track of all solutions solutions.append(x.copy()) # report progress print(‘>%d f(%s) = %.5f’ % (t, x, score)) return solutions |

We can then execute the search as before, and this time retrieve the list of solutions instead of the best final solution.

... # seed the pseudo random number generator seed(1) # define range for input bounds = asarray([[–1.0, 1.0], [–1.0, 1.0]]) # define the total iterations n_iter = 100 # steps size alpha = 0.007 # factor for average gradient beta1 = 0.9 # factor for average squared gradient beta2 = 0.99 # perform the gradient descent search with amsgrad solutions = amsgrad(objective, derivative, bounds, n_iter, alpha, beta1, beta2) |

We can then create a contour plot of the objective function, as before.

... # sample input range uniformly at 0.1 increments xaxis = arange(bounds[0,0], bounds[0,1], 0.1) yaxis = arange(bounds[1,0], bounds[1,1], 0.1) # create a mesh from the axis x, y = meshgrid(xaxis, yaxis) # compute targets results = objective(x, y) # create a filled contour plot with 50 levels and jet color scheme pyplot.contourf(x, y, results, levels=50, cmap=‘jet’) |

Finally, we can plot each solution found during the search as a white dot connected by a line.

... # plot the sample as black circles solutions = asarray(solutions) pyplot.plot(solutions[:, 0], solutions[:, 1], ‘.-‘, color=‘w’) |

Tying this all together, the complete example of performing the AMSGrad optimization on the test problem and plotting the results on a contour plot is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 |
# example of plotting the amsgrad search on a contour plot of the test function from math import sqrt from numpy import asarray from numpy import arange from numpy.random import rand from numpy.random import seed from numpy import meshgrid from matplotlib import pyplot from mpl_toolkits.mplot3d import Axes3D
# objective function def objective(x, y): return x**2.0 + y**2.0
# derivative of objective function def derivative(x, y): return asarray([x * 2.0, y * 2.0])
# gradient descent algorithm with amsgrad def amsgrad(objective, derivative, bounds, n_iter, alpha, beta1, beta2): solutions = list() # generate an initial point x = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] – bounds[:, 0]) # initialize moment vectors m = [0.0 for _ in range(bounds.shape[0])] v = [0.0 for _ in range(bounds.shape[0])] vhat = [0.0 for _ in range(bounds.shape[0])] # run the gradient descent for t in range(n_iter): # calculate gradient g(t) g = derivative(x[0], x[1]) # update variables one at a time for i in range(x.shape[0]): # m(t) = beta1(t) * m(t-1) + (1 – beta1(t)) * g(t) m[i] = beta1**(t+1) * m[i] + (1.0 – beta1**(t+1)) * g[i] # v(t) = beta2 * v(t-1) + (1 – beta2) * g(t)^2 v[i] = (beta2 * v[i]) + (1.0 – beta2) * g[i]**2 # vhat(t) = max(vhat(t-1), v(t)) vhat[i] = max(vhat[i], v[i]) # x(t) = x(t-1) – alpha(t) * m(t) / sqrt(vhat(t))) x[i] = x[i] – alpha * m[i] / (sqrt(vhat[i]) + 1e–8) # evaluate candidate point score = objective(x[0], x[1]) # keep track of all solutions solutions.append(x.copy()) # report progress print(‘>%d f(%s) = %.5f’ % (t, x, score)) return solutions
# seed the pseudo random number generator seed(1) # define range for input bounds = asarray([[–1.0, 1.0], [–1.0, 1.0]]) # define the total iterations n_iter = 100 # steps size alpha = 0.007 # factor for average gradient beta1 = 0.9 # factor for average squared gradient beta2 = 0.99 # perform the gradient descent search with amsgrad solutions = amsgrad(objective, derivative, bounds, n_iter, alpha, beta1, beta2) # sample input range uniformly at 0.1 increments xaxis = arange(bounds[0,0], bounds[0,1], 0.1) yaxis = arange(bounds[1,0], bounds[1,1], 0.1) # create a mesh from the axis x, y = meshgrid(xaxis, yaxis) # compute targets results = objective(x, y) # create a filled contour plot with 50 levels and jet color scheme pyplot.contourf(x, y, results, levels=50, cmap=‘jet’) # plot the sample as black circles solutions = asarray(solutions) pyplot.plot(solutions[:, 0], solutions[:, 1], ‘.-‘, color=‘w’) # show the plot pyplot.show() |

Running the example performs the search as before, except in this case, the contour plot of the objective function is created.

In this case, we can see that a white dot is shown for each solution found during the search, starting above the optima and progressively getting closer to the optima at the center of the plot.

This section provides more resources on the topic if you are looking to go deeper.

In this tutorial, you discovered how to develop gradient descent optimization with AMSGrad from scratch.

Specifically, you learned:

- Gradient descent is an optimization algorithm that uses the gradient of the objective function to navigate the search space.
- AMSGrad is an extension of the Adam version of gradient descent designed to accelerate the optimization process.
- How to implement the AMSGrad optimization algorithm from scratch and apply it to an objective function and evaluate the results.

**Do you have any questions?**

Ask your questions in the comments below and I will do my best to answer.

“If you dream of owning your own little slice of hell and turning it into a piece of heaven, then look no further!” So reads the real estate listing for a five-bedroom Colorado Springs house that’s been thoroughly vandalized by an evicted tenant. Accompanying the listing is an eight-minute YouTube video cataloguing the astonishing level of destruction inside the house—spray paint everywhere, feces and urine on the floor, dead cats in the bathroom. In the video, real estate agent Mimi Foster speculates that something suspicious happened with the tenant’s eviction, and pledges to dig into it further. I spoke to Foster—who had just finished leading a tour for local news cameras—about how to sell a house like this, the damage the housing crisis has done to landlord-tenant relations, her own side career as a romance novelist, and why her listing advises “DO NOT GO ON BACK DECK. DO NOT OPEN FREEZER IN BASEMENT.” Our conversation has been edited and condensed.

**You were just talking to news crews?**

My phone started ringing at 7 this morning and it hasn’t stopped. Yeah, KKTV was just here, the Denver Post, anyway … I hope good things are going to come out of this for my seller.

**What was it like when you first walked into this house?**

The first thing that you notice even before anything else is the smell. There’s a freezer that was full of meat that spoiled. I’ve been in there most of the day today, on and off, and—ugh, it’s horrible, we had to keep walking outside to breathe.

**Usually a video tour on a listing is meant to sell the attributes of a house, but yours seems to be telling a story. Why is that?**

I am an author. I write novels. And I’ve been doing real estate for about 30 years now. And I realized a long time ago that people want a story. And holy cow, this one has a story. I don’t want anybody coming in here expecting it to be a real home! It’s not! And it’s gonna take a hundred thousand to make it livable. Minimum. I wanted people to get engaged with the house. I have people calling me today saying “I’m not looking to buy a house, but can I come see this one?”

**You’re telling them the smell’s not worth it, right?**

I’m sitting in my car with the AC blowing straight at my face to get rid of the smell.

**Plus you had to be inside the house to make the video.**

I made that video a year ago and we just put it on the market yesterday.

**A year ago?**

We’ve been trying to get somebody to pay attention! This all happened right before COVID. The insurance company said, “Too bad, so sad, we don’t cover vandalism.” Pardon me, but—what the fuck?! What do you cover?

The house was going into foreclosure in the next two weeks. Banks have been on a foreclosure moratorium nationally. This house has been part of this moratorium, but it ended on June 30 and they said the owner only had two weeks. I said, “We are in the most insane market. If it’s ever gonna work, it’s gonna work now. Let’s just throw it on the market for what you owe on it.” Well, the bank contacted the owner last night and said that they were delaying foreclosure now. Is that because it’s gone viral? I don’t know.

**That’s remarkable.**

A year I’ve been dealing with this! It’s such a serious issue but it’s really funny, you know. You gotta laugh at it or you would just sit in the corner and cry.

**I found this whole story really sad.**

It’s tragic. It’s tragic on a lot of levels. The owner is sick, she lives out of state. She hasn’t seen the house in over ten years. The tenant was evicted in the fall of 2019. And there was a property manager then, and the property manager wrote the owner and said, the judge is going to let the tenant back in because she’s gonna go get her stuff, but she will be supervised. And so they let her back in, and it was in that few-hour window that all of the destruction happened.

**What happened to the supervision?**

Right? No court in the land would have let that woman back in there without supervision. So then the property manager called the owner and said, “There’s been some destruction.” She called me and said, I want you to list my house, and I was all excited, right? And then I found it.

**Oh no, were you the one who discovered the cats?**

Oh my god. It was awful. In the irony of ironies, this woman ran a pet shelter.

*Out of the house?*

Yes. So why did she lock the cats in the bathroom when she came back?

**You describe the situation as a landlord’s nightmare. You’ve also been a landlord. What do you think went wrong in this relationship?**

“Is it a wise investment? Don’t ask me that.”

I mean, I don’t have the tenant’s medical records, but she was crazy as a bedbug. The neighbors were afraid of her. When she found out that she was actually going to be evicted… the stuff on the wall is very personal against the owner. “EVICT THIS, BITCH,” and “WHO ARE YOU GONNA RENT THIS TO NOW, SUZY…” So definitely a personal attack, but I think it was because she was being evicted. She had no place to go.

**It seems like it’s sort of a ****housing crisis story**** as much as anything.**

She was evicted long before COVID. She hadn’t been paying rent in months. It never came down to that.

**OK, looking at the listing … Why can’t people go on the back deck?**

Because it’s falling off the house. The tenant probably had nothing to do with that. It’s just neglect. The house has been vacant for a year and a half.

**It seems sorta crazy to spend $600,000 on a house you wouldn’t even want to go in. Who’s the right buyer for this house?**

An investor who’s gonna fix it up and resell it to somebody who will live there. The 600K covers the mortgage that she’s behind on and the liens that are on the house.

**If this house is in good condition, what does it sell for, in this neighborhood?**

Realistically, 700. So is it a wise investment? Don’t ask me that.

**So the buyer needs to be adventurous.**

Yes! And quite frankly I was hoping that people would come out of the woodwork that wanted to help. Obviously this has gone viral. I haven’t seen any of it because I’ve been out all day.

**In the video you said, “Something smells worse than the odor in this house, and I intend to pursue it.” What are you looking for and have you found it?**

Let me tell you that, before you, I was on the phone with the property manager. She calls me and says, “Is this Mimi?” And she identifies herself, and I said, “Isn’t it amazing that I’ve been trying to call you for over a year and finally you’re returning a phone call.” And she said, “What makes you think that I ever had anything to do with this house?” And I said, “How about this stack of emails that I’ve got with your name on it?” She said, “I’ll see you in court, and the next time you hear from me it’ll be my attorney!” So … no, I don’t think I’ve found the answer yet.

“She said, ‘I’ll see you in court, and the next time you hear from me it’ll be my attorney!’”

**You’re an author, so you must admit there is real drama here.**

Real drama! I have all the emails! She doesn’t scare me.

**Tell me about the books you write.**

Steamy romances. Paranormal romances.

**Happy endings?**

*Always* happy endings.

**What’s the happy ending for this house?**

Oh, you’re good. It was a beautiful house. The happy ending is that the owner is made whole, and someone restores it, and a family moves in. That’s all I want.

Slate is covering the stories that matter to you. Become a Slate Plus member to support our work. Your first month is only $1.

Welcome to Northland Community & Technical College in Thief River Falls and East Grand Forks, MN.

Our Computer Service/Networking Associates degree offers students both theory and hands-on training in computer equipment servicing and networking. Computer skills development covers the hardware and software systems of current computer technology. Networking skills include switching, routing, server operating systems, directory services and much more. Many classes are built around specific industry certifications.

The program prepares graduates for immediate entry-level employment in any size company utilizing computer technology. Graduates adding industry certification such as A+, Network+, CCNA, etc. have an advantage.

The program provides students with the foundation required to build a rewarding career in the continually expanding field of computer service and networking.

EMPLOYMENT OPPORTUNITIES

Faster than average growth is expected in the next few years. Growth in networking of microcomputer systems in companies of every size offers more employment opportunities for the graduate.

The Microsoft certification should allow graduates to work in any size business. Opportunities are best for those who have both training and experience on a variety of systems, such as: Microsoft Windows, Novell NetWare, and UNIX. Graduates find positions as Computer Sales and Service Reps, Computer Service Helpdesk, Computer Service Technician.

source

EV battery swapping startup Ample has locked in two partnerships this month that will help fuel an expansion into Japan and New York City after years of working on the technology. The startup, which was founded in 2014 and came out of stealth in March, said Tuesday it has partnered with Japanese petroleum and energy company Eneos to jointly deploy and operate battery swapping infrastructure in Japan.

Over the next year, the two companies will pilot Ample’s fully automated swapping technology with a focus on ride-hailing, taxi, municipal, rental and last-mile delivery companies. Ample and Eneos will also evaluate whether swapping stations can offer other uses, such as a backup source of power for the energy grid. It is still early days for the partnership and few details have been disclosed; Ample, for instance didn’t share when the pilot program would begin or where in Japan it would initially launch. However, even with these scant details, Eneos’ interest signals that battery swapping — at least for Ample — is gaining some believers.

The Eneos announcement comes a few days after Ample launched a separate partnership with Sally, a New York City-based EV rental company for ride-hailing, taxi and last-mile deliveries. Ample and Sally will roll out five to 10 stations in NYC by the fourth quarter of this year, with plans to expand into other markets in 2021, according to Khaled Hassounah, founder and CEO of Ample.

Ample’s partnership with Sally will also expand to San Francisco in the next couple of months. Depending on the cost of using both services, this might be an advantageous deal for ride-hailing drivers in California, at least, where the state just decreed 90% of Uber and Lyft drivers must be in EVs by 2030.

“The goal is to ultimately make swapping stations as ubiquitous as gas stations,” Hassounah told TechCrunch.

Ample came out of stealth this March with five operational stations in the Bay Area and a partnership with Uber that entails drivers renting vehicles directly from Ample that have been retrofitted with the startup’s battery technology.

“The Ample architecture is designed to be integrated into any modern electric vehicle,” Levi Tillemann, VP for policy and international outreach at Ample, told TechCrunch. “Unlike a standard electric vehicle, where you have a battery pack that is never meant to be removed from the car, with the Ample system, you replace the battery pack with an adapter plate that essentially shares the exact same dimensions as the OEM designed battery pack. That adapter plate is the architecture that allows for battery swap.”

Ample’s standardized battery modules work with every vehicle that has been configured to run on the Ample platform, says Tillemann. With Ample’s partnership with Sally, the company will begin stepping away from running its own fleet, which it essentially launched to prove its business model. The company will work with Sally and probably other fleet and rental companies in the future to make vehicles Ample-enabled.

“Ample’s battery swapping works with any electric vehicle and dramatically reduces the cost and time it takes to install EV infrastructure by being a drop-in replacement for the OEM battery and does not require any modification to the car (either hardware or software),” said Hassounah.

One of the concerns that puts ride-hailing drivers off switching to EVs is the amount of time it takes to charge a battery. Hassounah says swapping a battery only takes 10 minutes, but that the company aims to reduce that to five minutes by the end of the year. Having a more efficient and seamless process might help ride-hailing drivers and logistics companies make the switch.

“Currently, drivers pay 10 cents per mile for swapping services, including energy, and the range varies based on the car model and battery size,” said Hassounah. “The price of the service varies depending on the price of electricity, but our goal is to have it be 10% to 20% cheaper than gas.”

When drivers want to swap a battery, they’ll use Ample’s app to find nearby stations and then initiate the autonomous swap. Each station can serve about five to six cars per hour, but the company expects to be able to serve double that by the end of the year. That said, this also depends on the amount of available power at a given site.

Tillemann says as Ample expands, the company aims to one day work with existing OEM partners to offer consumers the choice of having Ample production plates installed into their new vehicles on the production line.

“Our unit costs are very favorable for the battery swap systems,” he said. “They do not cost a lot to deploy, and that means, with a relatively low number of vehicles our battery swap architecture is economical and profitable.”

Eneos, which previously invested in Ample, according to the company, is committed to providing the next generation of energy supply. The company is also exploring hydrogen and recently partnered with Toyota’s Woven City, a futuristic prototype city that’s being built in Japan, to power the metropolis using hydrogen.

Gradient descent is an optimization algorithm that follows the negative gradient of an objective function in order to locate the minimum of the function.

A limitation of gradient descent is that it uses the same step size (learning rate) for each input variable. This can be a problem on objective functions that have different amounts of curvature in different dimensions, and in turn, may require a different sized step to a new point.

**Adaptive Gradients**, or **AdaGrad** for short, is an extension of the gradient descent optimization algorithm that allows the step size in each dimension used by the optimization algorithm to be automatically adapted based on the gradients seen for the variable (partial derivatives) seen over the course of the search.

In this tutorial, you will discover how to develop the gradient descent with adaptive gradients optimization algorithm from scratch.

After completing this tutorial, you will know:

- Gradient descent is an optimization algorithm that uses the gradient of the objective function to navigate the search space.
- Gradient descent can be updated to use an automatically adaptive step size for each input variable in the objective function, called adaptive gradients or AdaGrad.
- How to implement the AdaGrad optimization algorithm from scratch and apply it to an objective function and evaluate the results.

Let’s get started.

This tutorial is divided into three parts; they are:

- Gradient Descent
- Adaptive Gradient (AdaGrad)
- Gradient Descent With AdaGrad
- Two-Dimensional Test Problem
- Gradient Descent Optimization With AdaGrad
- Visualization of AdaGrad

Gradient descent is an optimization algorithm.

It is technically referred to as a first-order optimization algorithm as it explicitly makes use of the first order derivative of the target objective function.

First-order methods rely on gradient information to help direct the search for a minimum …

— Page 69, Algorithms for Optimization, 2019.

The first order derivative, or simply the “*derivative*,” is the rate of change or slope of the target function at a specific point, e.g. for a specific input.

If the target function takes multiple input variables, it is referred to as a multivariate function and the input variables can be thought of as a vector. In turn, the derivative of a multivariate target function may also be taken as a vector and is referred to generally as the “gradient.”

**Gradient**: First-order derivative for a multivariate objective function.

The derivative or the gradient points in the direction of the steepest ascent of the target function for a specific input.

Gradient descent refers to a minimization optimization algorithm that follows the negative of the gradient downhill of the target function to locate the minimum of the function.

The gradient descent algorithm requires a target function that is being optimized and the derivative function for the objective function. The target function *f()* returns a score for a given set of inputs, and the derivative function *f'()* gives the derivative of the target function for a given set of inputs.

The gradient descent algorithm requires a starting point (*x*) in the problem, such as a randomly selected point in the input space.

The derivative is then calculated and a step is taken in the input space that is expected to result in a downhill movement in the target function, assuming we are minimizing the target function.

A downhill movement is made by first calculating how far to move in the input space, calculated as the step size (called alpha or the learning rate) multiplied by the gradient. This is then subtracted from the current point, ensuring we move against the gradient, or down the target function.

- x = x – step_size * f'(x)

The steeper the objective function at a given point, the larger the magnitude of the gradient, and in turn, the larger the step taken in the search space. The size of the step taken is scaled using a step size hyperparameter.

**Step Size**(*alpha*): Hyperparameter that controls how far to move in the search space against the gradient each iteration of the algorithm.

If the step size is too small, the movement in the search space will be small and the search will take a long time. If the step size is too large, the search may bounce around the search space and skip over the optima.

Now that we are familiar with the gradient descent optimization algorithm, let’s take a look at AdaGrad.

The Adaptive Gradient algorithm, or AdaGrad for short, is an extension to the gradient descent optimization algorithm.

The algorithm was described by John Duchi, et al. in their 2011 paper titled “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.”

It is designed to accelerate the optimization process, e.g. decrease the number of function evaluations required to reach the optima, or to improve the capability of the optimization algorithm, e.g. result in a better final result.

The parameters with the largest partial derivative of the loss have a correspondingly rapid decrease in their learning rate, while parameters with small partial derivatives have a relatively small decrease in their learning rate.

— Page 307, Deep Learning, 2016.

A problem with the gradient descent algorithm is that the step size (learning rate) is the same for each variable or dimension in the search space. It is possible that better performance can be achieved using a step size that is tailored to each variable, allowing larger movements in dimensions with a consistently steep gradient and smaller movements in dimensions with less steep gradients.

AdaGrad is designed to specifically explore the idea of automatically tailoring the step size for each dimension in the search space.

The adaptive subgradient method, or Adagrad, adapts a learning rate for each component of x

— Page 77, Algorithms for Optimization, 2019.

This is achieved by first calculating a step size for a given dimension, then using the calculated step size to make a movement in that dimension using the partial derivative. This process is then repeated for each dimension in the search space.

Adagrad dulls the influence of parameters with consistently high gradients, thereby increasing the influence of parameters with infrequent updates.

— Page 77, Algorithms for Optimization, 2019.

AdaGrad is suited to objective functions where the curvature of the search space is different in different dimensions, allowing a more effective optimization given the customization of the step size in each dimension.

The algorithm requires that you set an initial step size for all input variables as per normal, such as 0.1 or 0.001, or similar. Although, the benefit of the algorithm is that it is not as sensitive to the initial learning rate as the gradient descent algorithm.

Adagrad is far less sensitive to the learning rate parameter alpha. The learning rate parameter is typically set to a default value of 0.01.

— Page 77, Algorithms for Optimization, 2019.

An internal variable is then maintained for each input variable that is the sum of the squared partial derivatives for the input variable observed during the search.

This sum of the squared partial derivatives is then used to calculate the step size for the variable by dividing the initial step size value (e.g. hyperparameter value specified at the start of the run) divided by the square root of the sum of the squared partial derivatives.

- cust_step_size = step_size / sqrt(s)

It is possible for the square root of the sum of squared partial derivatives to result in a value of 0.0, resulting in a divide by zero error. Therefore, a tiny value can be added to the denominator to avoid this possibility, such as 1e-8.

- cust_step_size = step_size / (1e-8 + sqrt(s))

Where *cust_step_size* is the calculated step size for an input variable for a given point during the search, *step_size* is the initial step size, *sqrt()* is the square root operation, and *s* is the sum of the squared partial derivatives for the input variable seen during the search so far.

The custom step size is then used to calculate the value for the variable in the next point or solution in the search.

- x(t+1) = x(t) – cust_step_size * f'(x(t))

This process is then repeated for each input variable until a new point in the search space is created and can be evaluated.

Importantly, the partial derivative for the current solution (iteration of the search) is included in the sum of the square root of partial derivatives.

We could maintain an array of partial derivatives or squared partial derivatives for each input variable, but this is not necessary. Instead, we simply maintain the sum of the squared partial derivatives and add new values to this sum along the way.

Now that we are familiar with the AdaGrad algorithm, let’s explore how we might implement it and evaluate its performance.

In this section, we will explore how to implement the gradient descent optimization algorithm with adaptive gradients.

First, let’s define an optimization function.

We will use a simple two-dimensional function that squares the input of each dimension and define the range of valid inputs from -1.0 to 1.0.

The *objective()* function below implements this function.

# objective function def objective(x, y): return x**2.0 + y**2.0 |

We can create a three-dimensional plot of the dataset to get a feeling for the curvature of the response surface.

The complete example of plotting the objective function is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# 3d plot of the test function from numpy import arange from numpy import meshgrid from matplotlib import pyplot
# objective function def objective(x, y): return x**2.0 + y**2.0
# define range for input r_min, r_max = –1.0, 1.0 # sample input range uniformly at 0.1 increments xaxis = arange(r_min, r_max, 0.1) yaxis = arange(r_min, r_max, 0.1) # create a mesh from the axis x, y = meshgrid(xaxis, yaxis) # compute targets results = objective(x, y) # create a surface plot with the jet color scheme figure = pyplot.figure() axis = figure.gca(projection=‘3d’) axis.plot_surface(x, y, results, cmap=‘jet’) # show the plot pyplot.show() |

Running the example creates a three-dimensional surface plot of the objective function.

We can see the familiar bowl shape with the global minima at f(0, 0) = 0.

We can also create a two-dimensional plot of the function. This will be helpful later when we want to plot the progress of the search.

The example below creates a contour plot of the objective function.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# contour plot of the test function from numpy import asarray from numpy import arange from numpy import meshgrid from matplotlib import pyplot
# objective function def objective(x, y): return x**2.0 + y**2.0
# define range for input bounds = asarray([[–1.0, 1.0], [–1.0, 1.0]]) # sample input range uniformly at 0.1 increments xaxis = arange(bounds[0,0], bounds[0,1], 0.1) yaxis = arange(bounds[1,0], bounds[1,1], 0.1) # create a mesh from the axis x, y = meshgrid(xaxis, yaxis) # compute targets results = objective(x, y) # create a filled contour plot with 50 levels and jet color scheme pyplot.contourf(x, y, results, levels=50, cmap=‘jet’) # show the plot pyplot.show() |

Running the example creates a two-dimensional contour plot of the objective function.

We can see the bowl shape compressed to contours shown with a color gradient. We will use this plot to plot the specific points explored during the progress of the search.

Now that we have a test objective function, let’s look at how we might implement the AdaGrad optimization algorithm.

We can apply the gradient descent with adaptive gradient algorithm to the test problem.

First, we need a function that calculates the derivative for this function.

The derivative of x^2 is x * 2 in each dimension.

The *derivative()* function implements this below.

# derivative of objective function def derivative(x, y): return asarray([x * 2.0, y * 2.0]) |

Next, we can implement gradient descent with adaptive gradients.

First, we can select a random point in the bounds of the problem as a starting point for the search.

This assumes we have an array that defines the bounds of the search with one row for each dimension and the first column defines the minimum and the second column defines the maximum of the dimension.

... # generate an initial point solution = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] – bounds[:, 0]) |

Next, we need to initialize the sum of the squared partial derivatives for each dimension to 0.0 values.

... # list of the sum square gradients for each variable sq_grad_sums = [0.0 for _ in range(bounds.shape[0])] |

We can then enumerate a fixed number of iterations of the search optimization algorithm defined by a “*n_iter*” hyperparameter.

... # run the gradient descent for it in range(n_iter): ... |

The first step is to calculate the gradient for the current solution using the *derivative()* function.

... # calculate gradient gradient = derivative(solution[0], solution[1]) |

We then need to calculate the square of the partial derivative of each variable and add them to the running sum of these values.

... # update the sum of the squared partial derivatives for i in range(gradient.shape[0]): sq_grad_sums[i] += gradient[i]**2.0 |

We can then use the sum squared partial derivatives and gradient to calculate the next point.

We will do this one variable at a time, first calculating the step size for the variable, then the new value for the variable. These values are built up in an array until we have a completely new solution that is in the steepest descent direction from the current point using the custom step sizes.

... # build a solution one variable at a time new_solution = list() for i in range(solution.shape[0]): # calculate the step size for this variable alpha = step_size / (1e–8 + sqrt(sq_grad_sums[i])) # calculate the new position in this variable value = solution[i] – alpha * gradient[i] # store this variable new_solution.append(value) |

This new solution can then be evaluated using the *objective()* function and the performance of the search can be reported.

... # evaluate candidate point solution = asarray(new_solution) solution_eval = objective(solution[0], solution[1]) # report progress print(‘>%d f(%s) = %.5f’ % (it, solution, solution_eval)) |

And that’s it.

We can tie all of this together into a function named *adagrad()* that takes the names of the objective function and the derivative function, an array with the bounds of the domain, and hyperparameter values for the total number of algorithm iterations and the initial learning rate, and returns the final solution and its evaluation.

This complete function is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# gradient descent algorithm with adagrad def adagrad(objective, derivative, bounds, n_iter, step_size): # generate an initial point solution = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] – bounds[:, 0]) # list of the sum square gradients for each variable sq_grad_sums = [0.0 for _ in range(bounds.shape[0])] # run the gradient descent for it in range(n_iter): # calculate gradient gradient = derivative(solution[0], solution[1]) # update the sum of the squared partial derivatives for i in range(gradient.shape[0]): sq_grad_sums[i] += gradient[i]**2.0 # build a solution one variable at a time new_solution = list() for i in range(solution.shape[0]): # calculate the step size for this variable alpha = step_size / (1e–8 + sqrt(sq_grad_sums[i])) # calculate the new position in this variable value = solution[i] – alpha * gradient[i] # store this variable new_solution.append(value) # evaluate candidate point solution = asarray(new_solution) solution_eval = objective(solution[0], solution[1]) # report progress print(‘>%d f(%s) = %.5f’ % (it, solution, solution_eval)) return [solution, solution_eval] |

**Note**: we have intentionally used lists and imperative coding style instead of vectorized operations for readability. Feel free to adapt the implementation to a vectorized implementation with NumPy arrays for better performance.

We can then define our hyperparameters and call the *adagrad()* function to optimize our test objective function.

In this case, we will use 50 iterations of the algorithm and an initial learning rate of 0.1, both chosen after a little trial and error.

... # seed the pseudo random number generator seed(1) # define range for input bounds = asarray([[–1.0, 1.0], [–1.0, 1.0]]) # define the total iterations n_iter = 50 # define the step size step_size = 0.1 # perform the gradient descent search with adagrad best, score = adagrad(objective, derivative, bounds, n_iter, step_size) print(‘Done!’) print(‘f(%s) = %f’ % (best, score)) |

Tying all of this together, the complete example of gradient descent optimization with adaptive gradients is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
# gradient descent optimization with adagrad for a two-dimensional test function from math import sqrt from numpy import asarray from numpy.random import rand from numpy.random import seed
# objective function def objective(x, y): return x**2.0 + y**2.0
# derivative of objective function def derivative(x, y): return asarray([x * 2.0, y * 2.0])
# gradient descent algorithm with adagrad def adagrad(objective, derivative, bounds, n_iter, step_size): # generate an initial point solution = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] – bounds[:, 0]) # list of the sum square gradients for each variable sq_grad_sums = [0.0 for _ in range(bounds.shape[0])] # run the gradient descent for it in range(n_iter): # calculate gradient gradient = derivative(solution[0], solution[1]) # update the sum of the squared partial derivatives for i in range(gradient.shape[0]): sq_grad_sums[i] += gradient[i]**2.0 # build a solution one variable at a time new_solution = list() for i in range(solution.shape[0]): # calculate the step size for this variable alpha = step_size / (1e–8 + sqrt(sq_grad_sums[i])) # calculate the new position in this variable value = solution[i] – alpha * gradient[i] # store this variable new_solution.append(value) # evaluate candidate point solution = asarray(new_solution) solution_eval = objective(solution[0], solution[1]) # report progress print(‘>%d f(%s) = %.5f’ % (it, solution, solution_eval)) return [solution, solution_eval]
# seed the pseudo random number generator seed(1) # define range for input bounds = asarray([[–1.0, 1.0], [–1.0, 1.0]]) # define the total iterations n_iter = 50 # define the step size step_size = 0.1 # perform the gradient descent search with adagrad best, score = adagrad(objective, derivative, bounds, n_iter, step_size) print(‘Done!’) print(‘f(%s) = %f’ % (best, score)) |

Running the example applies the AdaGrad optimization algorithm to our test problem and reports the performance of the search for each iteration of the algorithm.

**Note**: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, we can see that a near-optimal solution was found after perhaps 35 iterations of the search, with input values near 0.0 and 0.0, evaluating to 0.0.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
>0 f([-0.06595599 0.34064899]) = 0.12039 >1 f([-0.02902286 0.27948766]) = 0.07896 >2 f([-0.0129815 0.23463749]) = 0.05522 >3 f([-0.00582483 0.1993997 ]) = 0.03979 >4 f([-0.00261527 0.17071256]) = 0.02915 >5 f([-0.00117437 0.14686138]) = 0.02157 >6 f([-0.00052736 0.12676134]) = 0.01607 >7 f([-0.00023681 0.10966762]) = 0.01203 >8 f([-0.00010634 0.09503809]) = 0.00903 >9 f([-4.77542704e-05 8.24607972e-02]) = 0.00680 >10 f([-2.14444463e-05 7.16123835e-02]) = 0.00513 >11 f([-9.62980437e-06 6.22327049e-02]) = 0.00387 >12 f([-4.32434258e-06 5.41085063e-02]) = 0.00293 >13 f([-1.94188148e-06 4.70624414e-02]) = 0.00221 >14 f([-8.72017797e-07 4.09453989e-02]) = 0.00168 >15 f([-3.91586740e-07 3.56309531e-02]) = 0.00127 >16 f([-1.75845235e-07 3.10112252e-02]) = 0.00096 >17 f([-7.89647442e-08 2.69937139e-02]) = 0.00073 >18 f([-3.54597657e-08 2.34988084e-02]) = 0.00055 >19 f([-1.59234984e-08 2.04577993e-02]) = 0.00042 >20 f([-7.15057749e-09 1.78112581e-02]) = 0.00032 >21 f([-3.21102543e-09 1.55077005e-02]) = 0.00024 >22 f([-1.44193729e-09 1.35024688e-02]) = 0.00018 >23 f([-6.47513760e-10 1.17567908e-02]) = 0.00014 >24 f([-2.90771361e-10 1.02369798e-02]) = 0.00010 >25 f([-1.30573263e-10 8.91375193e-03]) = 0.00008 >26 f([-5.86349941e-11 7.76164047e-03]) = 0.00006 >27 f([-2.63305247e-11 6.75849105e-03]) = 0.00005 >28 f([-1.18239380e-11 5.88502652e-03]) = 0.00003 >29 f([-5.30963626e-12 5.12447017e-03]) = 0.00003 >30 f([-2.38433568e-12 4.46221948e-03]) = 0.00002 >31 f([-1.07070548e-12 3.88556303e-03]) = 0.00002 >32 f([-4.80809073e-13 3.38343471e-03]) = 0.00001 >33 f([-2.15911255e-13 2.94620023e-03]) = 0.00001 >34 f([-9.69567190e-14 2.56547145e-03]) = 0.00001 >35 f([-4.35392094e-14 2.23394494e-03]) = 0.00000 >36 f([-1.95516389e-14 1.94526160e-03]) = 0.00000 >37 f([-8.77982370e-15 1.69388439e-03]) = 0.00000 >38 f([-3.94265180e-15 1.47499203e-03]) = 0.00000 >39 f([-1.77048011e-15 1.28438640e-03]) = 0.00000 >40 f([-7.95048604e-16 1.11841198e-03]) = 0.00000 >41 f([-3.57023093e-16 9.73885702e-04]) = 0.00000 >42 f([-1.60324146e-16 8.48035867e-04]) = 0.00000 >43 f([-7.19948720e-17 7.38448972e-04]) = 0.00000 >44 f([-3.23298874e-17 6.43023418e-04]) = 0.00000 >45 f([-1.45180009e-17 5.59929193e-04]) = 0.00000 >46 f([-6.51942732e-18 4.87572776e-04]) = 0.00000 >47 f([-2.92760228e-18 4.24566574e-04]) = 0.00000 >48 f([-1.31466380e-18 3.69702307e-04]) = 0.00000 >49 f([-5.90360555e-19 3.21927835e-04]) = 0.00000 Done! f([-5.90360555e-19 3.21927835e-04]) = 0.000000 |

We can plot the progress of the search on a contour plot of the domain.

This can provide an intuition for the progress of the search over the iterations of the algorithm.

We must update the *adagrad()* function to maintain a list of all solutions found during the search, then return this list at the end of the search.

The updated version of the function with these changes is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# gradient descent algorithm with adagrad def adagrad(objective, derivative, bounds, n_iter, step_size): # track all solutions solutions = list() # generate an initial point solution = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] – bounds[:, 0]) # list of the sum square gradients for each variable sq_grad_sums = [0.0 for _ in range(bounds.shape[0])] # run the gradient descent for it in range(n_iter): # calculate gradient gradient = derivative(solution[0], solution[1]) # update the sum of the squared partial derivatives for i in range(gradient.shape[0]): sq_grad_sums[i] += gradient[i]**2.0 # build solution new_solution = list() for i in range(solution.shape[0]): # calculate the learning rate for this variable alpha = step_size / (1e–8 + sqrt(sq_grad_sums[i])) # calculate the new position in this variable value = solution[i] – alpha * gradient[i] new_solution.append(value) # store the new solution solution = asarray(new_solution) solutions.append(solution) # evaluate candidate point solution_eval = objective(solution[0], solution[1]) # report progress print(‘>%d f(%s) = %.5f’ % (it, solution, solution_eval)) return solutions |

We can then execute the search as before, and this time retrieve the list of solutions instead of the best final solution.

... # seed the pseudo random number generator seed(1) # define range for input bounds = asarray([[–1.0, 1.0], [–1.0, 1.0]]) # define the total iterations n_iter = 50 # define the step size step_size = 0.1 # perform the gradient descent search solutions = adagrad(objective, derivative, bounds, n_iter, step_size) |

We can then create a contour plot of the objective function, as before.

... # sample input range uniformly at 0.1 increments xaxis = arange(bounds[0,0], bounds[0,1], 0.1) yaxis = arange(bounds[1,0], bounds[1,1], 0.1) # create a mesh from the axis x, y = meshgrid(xaxis, yaxis) # compute targets results = objective(x, y) # create a filled contour plot with 50 levels and jet color scheme pyplot.contourf(x, y, results, levels=50, cmap=‘jet’) |

Finally, we can plot each solution found during the search as a white dot connected by a line.

... # plot the sample as black circles solutions = asarray(solutions) pyplot.plot(solutions[:, 0], solutions[:, 1], ‘.-‘, color=‘w’) |

Tying this all together, the complete example of performing the AdaGrad optimization on the test problem and plotting the results on a contour plot is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 |
# example of plotting the adagrad search on a contour plot of the test function from math import sqrt from numpy import asarray from numpy import arange from numpy.random import rand from numpy.random import seed from numpy import meshgrid from matplotlib import pyplot from mpl_toolkits.mplot3d import Axes3D
# objective function def objective(x, y): return x**2.0 + y**2.0
# derivative of objective function def derivative(x, y): return asarray([x * 2.0, y * 2.0])
# gradient descent algorithm with adagrad def adagrad(objective, derivative, bounds, n_iter, step_size): # track all solutions solutions = list() # generate an initial point solution = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] – bounds[:, 0]) # list of the sum square gradients for each variable sq_grad_sums = [0.0 for _ in range(bounds.shape[0])] # run the gradient descent for it in range(n_iter): # calculate gradient gradient = derivative(solution[0], solution[1]) # update the sum of the squared partial derivatives for i in range(gradient.shape[0]): sq_grad_sums[i] += gradient[i]**2.0 # build solution new_solution = list() for i in range(solution.shape[0]): # calculate the learning rate for this variable alpha = step_size / (1e–8 + sqrt(sq_grad_sums[i])) # calculate the new position in this variable value = solution[i] – alpha * gradient[i] new_solution.append(value) # store the new solution solution = asarray(new_solution) solutions.append(solution) # evaluate candidate point solution_eval = objective(solution[0], solution[1]) # report progress print(‘>%d f(%s) = %.5f’ % (it, solution, solution_eval)) return solutions
# seed the pseudo random number generator seed(1) # define range for input bounds = asarray([[–1.0, 1.0], [–1.0, 1.0]]) # define the total iterations n_iter = 50 # define the step size step_size = 0.1 # perform the gradient descent search solutions = adagrad(objective, derivative, bounds, n_iter, step_size) # sample input range uniformly at 0.1 increments xaxis = arange(bounds[0,0], bounds[0,1], 0.1) yaxis = arange(bounds[1,0], bounds[1,1], 0.1) # create a mesh from the axis x, y = meshgrid(xaxis, yaxis) # compute targets results = objective(x, y) # create a filled contour plot with 50 levels and jet color scheme pyplot.contourf(x, y, results, levels=50, cmap=‘jet’) # plot the sample as black circles solutions = asarray(solutions) pyplot.plot(solutions[:, 0], solutions[:, 1], ‘.-‘, color=‘w’) # show the plot pyplot.show() |

Running the example performs the search as before, except in this case, a contour plot of the objective function is created and a white dot is shown for each solution found during the search, starting above the optima and progressively getting closer to the optima at the center of the plot.

This section provides more resources on the topic if you are looking to go deeper.

In this tutorial, you discovered how to develop the gradient descent with adaptive gradients optimization algorithm from scratch.

Specifically, you learned:

- Gradient descent can be updated to use an automatically adaptive step size for each input variable in the objective function, called adaptive gradients or AdaGrad.
- How to implement the AdaGrad optimization algorithm from scratch and apply it to an objective function and evaluate the results.

**Do you have any questions?**

Ask your questions in the comments below and I will do my best to answer.

Democrats have long had a self-defeating desire to prove that they are America’s real party of fiscal responsibility, a pathology that has occasionally shined through as they’ve negotiated with Republicans over a potential infrastructure bill. Last month, for instance, a couple of GOP senators cracked the door open to paying for at least some of a bill with deficit spending—”is it something that is on the table? I think that’s probably accurate to say,” Alaska’s Lisa Murkowski said—only for Democrats to slam it back shut because they’d prefer to raise taxes on corporations and the rich. President Joe Biden, meanwhile, has himself said flatly that he’s “not willing to deficit spend.”

It will be a pity if Biden and the Democrats stick to that position. Whether they strike a deal with the GOP or decide to go it alone, insisting that every dollar of new road, bridge, or climate spending be paid is exactly the kind of self-torture that they ought to be avoiding. Moderates have already said they won’t go along with some of the tax hikes the White House has proposed to finance its broader agenda, which means that the way for Biden to avoid having to downsize his ambitions (what goes between pre-K and health care?) is to avoid promising to pay for everything in the first place.

Despite its supposed unpopularity, using deficit spending to fund big, one-time expenses like infrastructure upgrades and decarbonization actually makes economic sense. Even if you are a deficit hawk inclined to worry about the long-term impact of government debt, this is a completely reasonable instance to break out the nation’s credit card.

Part of the reason why is simply that interest rates are very low these days, which makes it a particularly good moment for the government to borrow and invest in projects that will spur the economy to grow faster in the long term (and, ideally, help the planet avoid climate catastrophe). The world’s investors are offering America a cheap line of credit if we want to make some much-needed upgrades to our roads, trains, and electric grid, and there’s no real reason to turn them down. Thanks to those rock bottom interest rates, the U.S. could add a trillion dollars or so to the debt right now without it making a massive difference to the longterm shape of the federal budget, which is mostly going to be determined by the costs of health care and Social Security. The federal debt held by the public is already around $22 trillion; bumping it up by 4 or 5 percent now won’t make or break anything. (Plus, the experience of heavy-borrowing Japan suggests the U.S. could possibly carry much higher debt without any catastrophic consequences.)

But there’s also any easy way for deficit-conscious lawmakers to choose a cautious middle ground that keeps the federal budget stable over the long-term while still using the government’s ability to borrow to its advantage. The trick is to pay for new infrastructure or climate spending with deficit spending up front, while raising taxes just enough to cover interest payments on the debt over time. It’d be a bit like if a car company decided to issue a bunch of bonds to fix up its production lines, and planned to just keep rolling them over while its profits grew.

To understand why that strategy makes sense, it helps to step back and think about what it means for the government to be “fiscally responsible.” Mainstream economists will tell you that it’s OK for the government to run a budget deficit, as long as the interest payments on its debt don’t start rising unsustainably high (you don’t want to end up in a situation where obligations to bond holders eat up a larger and larger share of the economy each year, until your only options are to pass draconian tax hikes, inflate away the debt, or default). As a result, it makes sense to pay for permanent programs that would otherwise add significantly to the deficit each year. But with big temporary expenses, like a war or historic decarbonization effort, it’s better to finance them with debt, then pay the cost down slowly over time. (This has been a mainstream idea in public finance since the late 1970s). One way to do that is to raise taxes slightly in order to cover the cost of interest, so that it doesn’t compound each year. As the years go on, the government’s revenues will grow with the economy, and both the principle and interest will become less of a burden, as they shrink compared to the country’s GDP.

I asked the Committee for a Responsible Federal Budget to calculate how much it would cost the government to just cover the annual interest payments on $1 trillion of new infrastructure or climate spending over the next two decades, assuming it didn’t compound. The grand total came out to $387 billion, over 20 years, or a modest $19 billion annually. (That’s about 2.6 percent of the current defense budget).

Another nice thing about this strategy is that Congress’s arcane rules also more or less encourage it. The budget reconciliation process, which Democrats will have to use to pass this bill if they don’t get bipartisan support, effectively requires that lawmakers cover the cost of permanent programs, but they don’t have to pay for temporary spending—that can add to the deficit. The only thing stopping Democrats from using debt smartly here is a misbegotten sense that the way to be responsible stewards of the budget is to avoid borrowing except in emergencies like COVID. In reality, you don’t have to be a deficit-phobe in order to be fiscally responsible. Democrats need to realize that before they shoot themselves in foot.

In this video, I share my Active Trader Pro day trading layout with suggestions that could improve your performance.

Topics include:

1. The five windows that should be in every layout.

2. Optimizing Orders, Positions, and Closed Positions windows for readability.

3. A look at the Directed Trade window with Level 2 / Times & Sales and shortcuts.

4. Placing critical windows logically near each other to improve response times.

5. Maximizing your screen “real estate”.

RESOURCES & LINKS

HONEST REVIEW FIDELITY | TRADING AND INVESTING

FIDELITY | FAST TRADES – ENTRY AND EXECUTION

FIDELITY | STOCK TRADE TRADE PRICE IMPROVEMENTS – HOW MUCH $$$ CAN YOU MAKE?

FIDELITY | TRADE ARMOR – TUTORIAL & REVIEW

source

Jaguar Land Rover is developing a hydrogen fuel cell vehicle based on the new Defender SUV and plans to begin testing the prototype next year.

The prototype program, known as Project Zeus, is part of JLR’s larger aim to only produce zero-tailpipe emissions vehicles by 20236. JLR has also made a commitment to have zero carbon emissions across its supply chain, products and operations by 2039.

Project Zeus is partially funded by the UK government-backed Advanced Propulsion Center. The automaker has also tapped AVL, Delta Motorsport, Marelli Automotive Systems and the UK Battery Industrialization Center to help develop the prototype. The testing program is designed to help engineers understand how a hydrogen powertrain can be developed that would meet the performance and capability (like towing and off roading) standards that Land Rover customers expect.

Fuel cells combine hydrogen and oxygen to produce electricity without combustion. The electricity generated from hydrogen is used to power an electric motor. Some automakers, researchers and policymakers have advocated for the technology because hydrogen-powered FCEVs can be refueled quickly, have a high-energy density and don’t lose as much range in cold temperatures. The combination means EVs that can travel longer distances.

Few fuel cell EVs, otherwise known as FCEVs, are on the market today in part because of a lack of refueling stations. The Toyota Mirai is one example.

Data from the International Energy Agency and recent commitments by automakers suggests that might be changing. Last month, BMW Chairman Oliver Zipse said the automaker plans to produce a small number of hydrogen fuel-cell powered X5 SUVs next year.

The number of FCEVs in the world nearly doubled to 25,210 units in 2019 from the previous year, the latest data from the IEA shows. The United States has been the leader in sales, although there was a dip in 2019, followed by China, Japan and Korea.

Japan has been a leader on the infrastructure end as it aims to have 200,000 FCEVs on the road by 2025. The country had installed 113 stations as of 2019, nearly twice as many as the United States.

“We know hydrogen has a role to play in the future powertrain mix across the whole transport industry, and alongside battery electric vehicles, it offers another zero tailpipe emission solution for the specific capabilities and requirements of Jaguar Land Rover’s world class line-up of vehicles,” Ralph Clague, the head of hydrogen and fuel cells for Jaguar Land Rover said in a statement.