Yes, you are not required to use MUI at all if you product doesn't need it.
We highly recommend using it for things like forms and buttons, etc. But if you don't need it, that is up to you. Many customers use custom components for everything they do. Also, if you have your own design system and components, you can also intergrate that with Quest as well copying and pasting placeholder components you can replace with your own components later.
Yes, by using Auto layout you can easily set how a component should behave relative to the parent or browser’s size. If you want to create different layouts or looks for different device screen sizes, you can simply create a variant for that and then use bindings to display which variant version you want for which screen size you need it for.
Yes, you can nest unlimited amount of components inside your designs. For instance, if you have a Main component in Figma that is your custom component you want to make a React component but inside that Figma component you have an instance of another component, you can import that into Quest and Quest will see the link to that component instance and treat it as such.
You are not required to import components in any order. You can import that parent first then the child or vice versa. The link to the component instance will exist even if the component is not yet in Quest.