$\begingroup$
Choosing the right motor can be a challenge. Unfortunately, the short answer to your question is no, there is not a simple formula for choosing a motor for a UAV. It's going to take some research regardless. This question is fairly open-ended, so I'll try to cover a few approaches.
The first thing you need to identify is the purpose of your craft. Are you looking to lift and transport a payload, are you interested in smooth controlled flight for aerial photography, or are you wanting to go fast and have immediate control and responsiveness? Each purpose for a multi-rotor is going to have extremely different approaches to the power train.
Part of what you need to analyze is how important several key variables are to your needs. The core needs are flight time (efficiency), speed, and handling. Where your needs fit along those spectrums will impact your motor choices.
The variables in the powertrain that correlate will be motor stator size and performance design characteristics, prop size, and battery voltage and capacity. Sorry nice this question was specific to motors I'll leave off the battery capacity component, but voltage is related to motor selection, so we do have to discuss the battery to a certain extent in conjunction with the motor choice.
There are a few rules of thumb when it comes to powertrain selection.
The more weight you need to lift, the larger the prop you will need to lift it efficiently. Essentially the loading of the effective disc area of the prop is proportional to efficiency. The larger the prop also the larger the motor that will be needed to be able to spin it effectively and the lower the relative Kv of the motor.
There is also an inverse relationship between the size of the motor and the efficiency of the motor. If you're going for efficiency, you'll want to use the smallest motor that can effectively spin the size prop needed for your payload weight. If you're going for maximized lift capacity, control, smoothness or immediacy of response, you'll want to find a balance more towards a larger motor that is somewhat under-propped.
The larger the motor the more torque it is capable of producing, so going up in stator size and other motor performance metrics such as magnet strength, narrower air-gaps between the stator and the magnets, lower resistance windings, all play a role in the level of control the multi-rotor is able to enforce. In general, the higher the torque of the motor, the more control the motor can exert over a given prop, both in terms of the maximum rate of change of RPMs and the maximum reachable RPMs as a percentage of the no-load RPMs of the motor (typically defined as Kv times the applied voltage)
It's important to note that when it comes to multi-rotor UAVs, maximum static thrust is not the only (or even necessarily the primary) indicator of how suitable a motor is for a given application. The more important component to consider is actually how quickly the motor can change the RPMs of the propeller. If your primary goals are heavy lifting for the physical size (high disc loading) then it is much closer to static load conditions, and maximum thrust is a much better metric. In any case, where you are looking for smoothness or control, the rate of change is more important than the static output. There is often a relationship between high static thrust and torque, however, which is why many freestyle and race quads are often a 10:1 or greater power to weight ratio even if they rarely use the full range of the throttle. Higher torque means more control and more immediate response beyond just maximum speed. It also means more authority for the PID controllers that are being used to stabilize and direct the craft. However a larger motor with a lower Kv and lower thrust can sometimes produce better results than a smaller higher Kv motor that produces more thrust bust changes RPMs much less quickly, so some fine-tuning is necessary beyond the simple thrust to weight calculations.
It's also important to note that thrust and speed don't always equate either. If you're looking to go fast, you will often get more speed out of a bi-blade prop that generates less thrust than a tri-blade that generates more thrust but spins at lower RPMs. This has to do with the way the pitch of the prop moves through the air as the craft moves forward. Also, a large low Kv motor that generates a lot of static thrust will not necessarily be faster than a smaller higher Kv motor that generates less static thrust but achieves higher RPMs in the air. Even on the same prop the lower Kv motor often will not be as fast due to the differences between static and dynamic loads (varying angle of attack). The larger lower Kv motor may hit higher RPMs under static load due to the higher torque, but the lower torque but higher Kv motor may hit higher RPMs when the load reduces in forward flight.
Voltage is another critical component as it impacts the motor Kv range that is applicable for a given prop size. Choosing a voltage is also dependent on your application. Higher voltage is typically needed when disc loading (lift capacity), power consumption needs, or speed needs increase. If efficiency is a goal, lower voltages should be considered. There is a lot more to that discussion in terms of the electrical losses vs the battery chemistry gains of different cell counts under different applications, but that is a bit outside the scope of this question, so I'll merely say that the higher the voltage chosen, the lower the Kv for a given prop size, and the lower the voltage the higher the Kv. You can calculate the conversion by simple multiplication, if you have a combination you know works well on 4 cells and you want to switch to a 6 cell lipo, simply multiply the existing Kv by 3 the. Divide the answer by 6 to get the equivalent Kv at 6 cells. You can also use the nominal voltage of the batteries as well.
(Kve * Ve)/Vt = Kvt
Where:
Kve is the existing Kv,
Ve is the existing voltage (or cell count),
Vt is the target voltage (or cell count),
Kvt is the target Kv
Generally speaking, the manufacturer specifications will have a list of props that are suitable for a given motor, but often the sheets are not very specific, and their thrust and efficiency charts are all over the place in terms of comparable data. If possible find a third part testing website like www.miniquadtestbench.com, EngineerX on YouTube, or the Micro Motor Test Lab on Facebook. This will be extremely helpful in identifying the range of motors and props that might work well in your application. From there you can weigh the variables I've mentioned and make adjustments. If you can't find independent data, take the manufacturer data with a grain of salt as a general ballpark figure.
Again, as mentioned elsewhere, this is all very specific to multi-rotor craft. Fixed-wing has very different considerations that are related to things like wrong loading, angle of attack, and airspeed, though some of the principles I mentioned do still apply. That's all a bit out of my wheelhouse though, so I'll leave that answer to someone with more experience and expertise in fixed-wing power systems.