We do a lot of Sencha+OpenACS work, so here are a few notes/suggestions:
-- Sencha Touch has some issues with many HTC android devices and many pre 2.2 android versions with things like scrolling performance. It also doesn't work (although I haven't tested - they don't claim support and I'm pretty sure it would fail to work) on Windows 7 phones (although the market share is tiny, Gartner thinks they'll pick up steam and Nokia will probably give them a big boost) nor several custom browsers (opera/firefox mobile) vs the native webkits on iphone/android. If you are fine with 85% of the mobile market in compatibility, then great - otherwise you would want to use jquery mobile.
Remember, both can take advantage of json data from a RESTful API, and you have to have a different front-end anyways for mobile, so its possible to use jquery mobile but Sencha's ExtJS.
-- I think it will probably be difficult to use openACS but not aolserver or at least not worth the trouble. It seems like switching from both at once would be less painful. What does aolserver do that you don't like?
-- If you want a really tiny REST interface that returns JSON, PHP is probably the most obvious choice, but you could also look at node.js (its javascript on the back end) as it makes that very convenient.
-- start your forms in Extjs 4.0 (vs 3.X) since they have trashed the form layout (now you just use regular layouts like vbox to position elements), as well as graphing being migrated to HTML5 from flash.
Your note here:
"- There is a smooth migration path with little overhead: We start with rewriting the most important pages in Sencha while maintaining the HTML GUI for the more complex cases. Later we could get rid of the HTML GUI completely."
Is exactly what we did. Its too big to replace openACS/aolserver, re-write all the code, etc. at once.