Double backwards

PyTorch Developer Podcast

Player FM - Internet Radio Done Right

32 subscribers

Toegevoegd vier jaar geleden

Inhoud geleverd door PyTorch, Edward Yang, and Team PyTorch. Alle podcastinhoud, inclusief afleveringen, afbeeldingen en podcastbeschrijvingen, wordt rechtstreeks geüpload en geleverd door PyTorch, Edward Yang, and Team PyTorch of hun podcastplatformpartner. Als u denkt dat iemand uw auteursrechtelijk beschermde werk zonder uw toestemming gebruikt, kunt u het hier beschreven proces https://nl.player.fm/legal volgen.

Young and Profiting with Hala Taha (Entrepreneurship, Sales, Marketing)

1
Reid Hoffman: LinkedIn Co-Founder on Building and Scaling Massively Valuable Companies Fast | Entrepreneurship | E332 51:40

11 weeks geleden51:40

Later Afspelen

Lijsten

Vind ik leuk

Leuk

51:40

Despite having a strong product idea, Reid Hoffman’s first startup collapsed, forcing him to return investors’ capital. This tough experience reshaped his approach to entrepreneurship. By embracing failure, iterating quickly, and adapting relentlessly, he went on to become a leader at PayPal and later, the co-founder of LinkedIn. In this episode, Reid shares the concept of blitzscaling, which prioritizes speed over perfection, smart strategies for taking risks, and insights on achieving rapid market dominance. In this episode, Hala and Reid will discuss: (00:00) Introduction (01:32) Building Impact-Driven Businesses (02:56) Why We Need More Entrepreneurs (04:31) The Vision Behind LinkedIn’s Success (06:43) Lessons from a Failed Startup (09:26) Making Quick, Intense Decisions at PayPal (12:39) Blitzscaling: Prioritizing Speed Over Efficiency (18:10) Maintaining Company Culture While Scaling (21:20) The Power of Early Market Dominance (25:01) The Five Stages of Company Growth (28:54) Strategies for Taking Intelligent Risks (31:44) Why Product Perfection Delays Success (33:25) Pivoting Early to Seize New Opportunities (36:18) Entrepreneurship as a Team Sport Reid Hoffman is an entrepreneur, investor, partner at Greylock, and co-founder of LinkedIn and Inflection AI. He was an executive at PayPal and a founding investor in several companies, including OpenAI. Reid actively supports various non-profits and has received numerous accolades, including an honorary CBE from the Queen of England and the Salute to Greatness Award from the Martin Luther King Jr. Center for his philanthropic efforts. Resources Mentioned: Reid’s Book, Blitzscaling: The Lightning-Fast Path to Building Massively Valuable Companies : amzn.to/4jnQkfQ Sponsored By: OpenPhone - Get 20% off 6 months at openphone.com/PROFITING Shopify - Sign up for a one-dollar-per-month trial period at youngandprofiting.co/shopify Airbnb - Your home might be worth more than you think. Find out how much at airbnb.com/host Rocket Money - Cancel your unwanted subscriptions and reach your financial goals faster with Rocket Money. Go to rocketmoney.com/profiting Indeed - Get a $75 job credit at indeed.com/profiting RobinHood - Receive your 3% boost on annual IRA contributions, sign up at robinhood.com/gold Active Deals - youngandprofiting.com/deals Key YAP Links Reviews - ratethispodcast.com/yap Youtube - youtube.com/c/YoungandProfiting LinkedIn - linkedin.com/in/htaha/ Instagram - instagram.com/yapwithhala/ Social + Podcast Services: yapmedia.com Transcripts - youngandprofiting.com/episodes-new All Show Keywords: Entrepreneurship, entrepreneurship podcast, Business, Business podcast, Self Improvement, Self-Improvement, Personal development, Starting a business, Strategy, Investing, Sales, Selling, Psychology, Productivity, Entrepreneurs, AI, Artificial Intelligence, Technology, Marketing, Negotiation, Money, Finance, Side hustle, Startup, mental health, Career, Leadership, Mindset, Health, Growth mindset. Career, Success, Entrepreneurship, Productivity, Careers, Startup, Entrepreneurs, Business Ideas, Growth Hacks, Career Development, Money Management, Opportunities, Professionals, Workplace, Career podcast, Entrepreneurship podcast…

4 jaar geleden 16:39

MP3•Thuis aflevering

Double backwards is PyTorch's way of implementing higher order differentiation. Why might you want it? How does it work? What are some of the weird things that happen when you do this?

Further reading.

Epic PR that added double backwards support for convolution initially https://github.com/pytorch/pytorch/pull/1643

83 afleveringen

#Tech #PyTorch #Edward Yang #Team PyTorch #Deep Learning #Machine Learning

Double backwards

PyTorch Developer Podcast

32 subscribers

published 4 jaar geleden

MP3•Thuis aflevering

Double backwards is PyTorch's way of implementing higher order differentiation. Why might you want it? How does it work? What are some of the weird things that happen when you do this?

Further reading.

Epic PR that added double backwards support for convolution initially https://github.com/pytorch/pytorch/pull/1643

83 afleveringen

#Tech #PyTorch #Edward Yang #Team PyTorch #Deep Learning #Machine Learning

Alle afleveringen

1
Compiler collectives 16:33

36 weeks geleden16:33

16:33

Compiler collectives are a PT2 feature where by compiler instances across multiple ranks use NCCL collectives to communicate information to other instances. This is used to ensure we consistently decide if inputs or static or dynamic across all ranks. See also PR at https://github.com/pytorch/pytorch/pull/130935…

1
TORCH_TRACE and tlparse 15:28

50 weeks geleden15:28

15:28

TORCH_TRACE and tlparse are a structured log and log parser for PyTorch 2. It gives useful information about what code was compiled and what the intermediate build products look like.

1
Higher order operators 17:10

51 weeks geleden17:10

17:10

Higher order operators are a special form of operators in torch.ops which have relaxed input argument requirements: in particular, they can accept any form of argument, including Python callables. Their name is based off of their most common use case, which is to represent higher order functions like control flow operators. However, they are also used to implement other variants of basic operators and can also be used to smuggle in Python data that is quite unusual. They are implemented using a Python dispatcher.…

1
Inductor - Post-grad FX passes 24:07

1 year geleden24:07

24:07

The post-grad FX passes in Inductor run after AOTAutograd has functionalized and normalized the input program into separate forward/backward graphs. As such, they generally can assume that the graph in question is functionalized, except for some mutations to inputs at the end of the graph. At the end of post-grad passes, there are special passes that reintroduce mutation into the graph before going into the rest of Inductor lowering which is generally aware of passes. The post-grad FX passes are varied but are typically domain specific passes making local changes to specific parts of the graph.…

1
CUDA graph trees 20:50

1 year geleden20:50

20:50

CUDA graph trees are the internal implementation of CUDA graphs used in PT2 when you say mode="reduce-overhead". Their primary innovation is that they allow the reuse of memory across multiple CUDA graphs, as long as they form a tree structure of potential paths you can go down with the CUDA graph. This greatly reduced the memory usage of CUDA graphs in PT2. There are some operational implications to using CUDA graphs which are described in the podcast.…

1
Min-cut partitioner 15:56

1 year geleden15:56

15:56

The min-cut partitioner makes decisions about what to save for backwards when splitting the forward and backwards graph from the joint graph traced by AOTAutograd. Crucially, it doesn't actually do a "split"; instead, it is deciding how much of the joint graph should be used for backwards. I also talk about the backward retracing problem.…

1
AOTInductor 17:30

1 year geleden17:30

17:30

AOTInductor is a feature in PyTorch that lets you export an inference model into a self-contained dynamic library, which can subsequently be loaded and used to run optimized inference. It is aimed primarily at CUDA and CPU inference applications, for situations when your model export once to be exported once while your runtime may still get continuous updates. One of the big underlying organizing principles is a limited ABI which does not include libtorch, which allows these libraries to stay stable over updates to the runtime. There are many export-like use cases you might be interested in using AOTInductor for, and some of the pieces should be useful, but AOTInductor does not necessarily solve them.…

1
Tensor subclasses and PT2 13:25

1 year geleden13:25

13:25

Tensor subclasses allow you to add extend PyTorch with new types of tensors without having to write any C++. They have been used to implement DTensor, FP8, Nested Jagged Tensor and Complex Tensor. Recent work by Brian Hirsh means that we can compile tensor subclasses in PT2, eliminating their overhead. The basic mechanism by which this compilation works is a desugaring process in AOTAutograd. There are some complications involving views, dynamic shapes and tangent metadata mismatch.…

1
Compiled autograd 18:07

1 year geleden18:07

18:07

Compiled autograd is an extension to PT2 that permits compiling the entirety of a backward() call in PyTorch. This allows us to fuse accumulate grad nodes as well as trace through arbitrarily complicated Python backward hooks. Compiled autograd is an important part of our plans for compiled DDP/FSDP as well as for whole-graph compilation.…

1
PT2 extension points 15:54

1 year geleden15:54

15:54

We discuss some extension points for customizing PT2 behavior across Dynamo, AOTAutograd and Inductor.

1
Inductor - Define-by-run IR 12:06

1 year geleden12:06

12:06

Define-by-run IR is how Inductor defines the internal compute of a pointwise/reduction operation. It is characterized by a function that calls a number of functions in the 'ops' namespace, where these ops can be overridden by different handlers depending on what kind of semantic analysis you need to do. The ops Inductor supports include regular arithmetic operators, but also memory load/store, indirect indexing, masking and collective operations like reductions.…

1
Unsigned integers 13:07

1 year geleden13:07

13:07

Traditionally, unsigned integer support in PyTorch was not great; we only support uint8. Recently, we added support for uint16, uint32 and uint64. Bare bones functionality works, but I'm entreating the community to help us build out the rest. In particular, for most operations, we plan to use PT2 to build anything else. But if you have an eager kernel you really need, send us a PR and we'll put it in. While most of the implementation was straightforward, there are some weirdnesses related to type promotion inconsistencies with numpy and dealing with the upper range of uint64. There is also upcoming support for sub-byte dtypes uint1-7, and these will exclusively be implemented via PT2.…

1
Inductor - IR 18:00

1 year geleden18:00

18:00

Inductor IR is an intermediate representation that lives between ATen FX graphs and the final Triton code generated by Inductor. It was designed to faithfully represent PyTorch semantics and accordingly models views, mutation and striding. When you write a lowering from ATen operators to Inductor IR, you get a TensorBox for each Tensor argument which contains a reference to the underlying IR (via StorageBox, and then a Buffer/ComputedBuffer) that says how the Tensor was computed. The inner computation is represented via define-by-run, which allows for compact definition of IR representation, while still allowing you to extract an FX graph out if you desire. Scheduling then takes buffers of inductor IR and decides what can be fused. Inductor IR may have too many nodes, this would be a good thing to refactor in the future.…

4 years geleden18:00

18:00

In this episode I talk about reduced precision floating point formats float16 (aka half precision) and bfloat16. I'll discuss what floating point numbers are, how these two formats vary, and some of the practical considerations that arise when you are working with numeric code in PyTorch that also needs to work in reduced precision. Did you know that we do all CUDA computations in float32, even if the source tensors are stored as float16? Now you know! Further reading. The Wikipedia article on IEEE floating point is pretty great https://en.wikipedia.org/wiki/IEEE_754 How bfloat16 works out when doing training https://arxiv.org/abs/1905.12322 Definition of acc_type in PyTorch https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/AccumulateType.h…

Welkom op Player FM!

Player FM scant het web op podcasts van hoge kwaliteit waarvan u nu kunt genieten. Het is de beste podcast-app en werkt op Android, iPhone en internet. Aanmelden om abonnementen op verschillende apparaten te synchroniseren.