Reason About A Problem In The Age Of LLMs & Automation
These are my personal thoughts based on my experience of working in an multi-disciplinary lab, we all are divided into two groups- Science and Engineering.
Science group is responsible for studying principles of material science, chemistry, human brain(neuroscience), device physics and applying them to fabricate molecular memristors based IMC(In-memory computing) chip, after fabrication, these chips go for the "characterization", "testing" and finally supplied to the engineering group. I am a part of Engineering group where we build systems using these chips, basically in layman's terms, we are responsible for making this chip "useful" for the real world applications. My colleagues in this group are from digital/analog electronics, electrical engineering background and myself from computer science engineering.
Reasoning is not about solving fast or knowing the right formula - it is all about "how you think" #
We can start by understanding the problem space-
- What is actually being asked?
- What is known vs unknown?
- What are the constraints(physics, data, cost, time, biology, hardware limits)?
Most people don’t care about these questions and rush to the existing tools to solve their problems ASAP, which won't help if your problem is still in early stages and quite vague and requires you to break into multi-disciplinary realms.
Reducing the problems to first principles, instead of saying “this is a hardware problem/ML problem /Biology problem” #
We should ask:
- What is flowing? (data/energy/signals/material)
For example: In analog domain, we deal with inputs and outputs in the form of voltages, current and conductance values. In digital, we have a discrete world where the continous analog values are mapped into bits with fixed size which we refer to as "signal". In machine learning, the inputs and the outputs are the data provided during the training and inference stages. The property of the material decides how a chip interacts with the physical world- noise and temperature.
- What transforms? (computation/reactions/force)
For example: In analog, the computation(MAC operation) consist of multiplication that is done using Ohm's law and the accumulation by Kirchhoff's Current Law (KCL). You can study the equations to understand how a memristor is able to attain 14 states/14 bits in digital by studying the equations of the device based on this material. By studying the equations, you will see how different physical variables like thermodynamic states and forces are involved and how these variables affect the final output.
- What is conserved or limited?(power,memory,entropy,mass)
For example: How the material affects the power consumption and energy requirements?
This way we can jump domains because first principles don’t change.
Building a rough internal mental model #
- If I change X, what happens to Y?
- Where are the bottlenecks?
- What assumptions we are making?
Keep updating the model with time as your understanding gets better.
Decomposing without compartmentalizing #
For example: If I am working on molecular memristor based in-memory computing hardware for ML models. I know hardware, algo, physics, material, each of these influences the other. We can treat them as different interfaces not a domain restrictions.
- Why does this fail under these conditions?
- What would happen if this assumption breaks?
- What does the system do at extremes?
These are the good questions that can move problems further even without any answers yet.
For example: I did my undergrad in CS but now I am hacking molecular electronics to run energy efficient ML models.
- Why to care about energy efficiency?
- Why to care about building a new hardware based on a new material? Silicon isn't enough?
- Why to hack molecular electronics - to attain multiple states ? Digital electronics(binary states 0s and 1s) not enough?
- What kind of applications/requirements inspires us to experiment with a new science?
If I have to run any algo on a new architecture, first, I will need to understand:
- How traditional algo work on traditional arch?
- What if I change the arch and replace it with bio-inspired arch? What changes needs to be done to the compiler?
- Or do I also need to change the algo?
- How memory hierarchy looks like for a new arch based on IMC chip?
- Is there any limit to this memory?
- What if I want to run a baremetal program targetting a RRAM(memristor) based IMC chip?
I'll have to figure out what all I need to run a program without any OS support. I will study about linkerscripts, how a program section and data section are mapped to the physical memory addresses directly without any intervention of register file or w/o any virtual memory.
Mentality that helps here: “I don't know enough yet but I can learn enough to move forward.”
We just can’t quit the problem if it requires chem, bio or math that we havent seen before. Navigating across domains without being scared of unfamiliar territory is a skill of this age where the generalists are thriving and specialists are getting obsolete.
- Previous: Ideas on low bit architectures