The path to self-flying aircraft & taxis

Flying an aircraft as a human pilot is bound to go the way of horseback riding. One will be able to pursue it as a hobby or sport — in every other case, we will be flying in aircraft and taking airborne taxi flights without a human pilot on board.

But how will we get there?

Below is an overview of that, taking industry trends into account, and focusing mostly on Commercial Air Transport — when people are transported through air for a fee. Differences to private flight will be highlighted at a few points below.

Why do we not want humans to fly our aircraft?

As with all undertakings, it is crucial to ask: “why?”

The answer is primarily cost, then safety, but it’s a bit more complex than that.

Currently a commercial air carrier’s HR budget is about 5% of it’s overall annual budget, which one would think is not that much — why would one spend all the effort of removing only 5% of expenses? Still, especially since 2001, air carriers are in a wage war against air crew, with ‘pay to fly’ schemes (yes, sometimes the pilot flying you is paying more than you did to be there), zero hour contracting, pilots not employed as regular employees but sub-contracted, wages hitting rock bottom. The reason they do this is that all other expenses have already been optimized to the fullest extent, with ongoing optimization taking place in the long term — for example fuel burn is being reduced by 1.3% per year on average, and has reduced by 45% between 1968 and 2014. A current passenger aircraft consumes less than 3 liters of fuel per passenger per 100 km, comparable or less than your car, while transporting you at 800+ km/h.

Another aspect of the human pilot is that there is a shortage of them. This seems contradictory in light of the deteriorating pilot employee and wage conditions, as one would assume lack of supply results in rising costs and improving conditions, but for some reason this is not the case. For the aviation industry to grow more rapidly, this pilot shortage is a bottleneck that can be removed by reducing the number of pilots on board and then removing them altogether.

Enter flying taxis: aircraft that carry passengers, but only have 2–4 seats in total, maybe 6, and expected to be deployed in the tens of thousands per year in a few years time. These aircraft being electrically driven, are also extremely cost efficient compared to traditional aircraft, where the fuel cost is significant. A human pilot takes up a seat and reduces the passenger capacity by 25–50%, adds a significant cost to the flight and will not be available at desired numbers when the time comes. For the flying taxi industry to scale according to expectations, these aircraft have to be autonomous, with no human pilot on board.

Aviation is bound for explosive growth, but it cannot do that if it depends on human pilots.

On top of this all is a safety aspect: over 60% of aviation accidents are due to pilot error. Automation will do better.

What does a self-flying aircraft need to able to do?

Looking at a commercial flight, where the departure and destination are known prior to flying, in a positive scenario, on a brush stroke level the following is needed:

• a route that will be flown has to be planned
• passengers boarded
• taxi on ground & take off
• fly en route
• land & taxi on ground
• passengers disembarked

The above would suffice in a perfect world and if there were was nothing ever in the air, no weather, no other aircraft, no birds or similar. This would miss the point of commercial air transport, which preferably flies in every kind of weather and there are a lot of these flights taking place at the same time. Thus the following is needed too:

• flight coordination with other aircraft
• airborne obstacle detection & avoidance
• handling of unusual & emergency situations

On top of all this, what is often forgotten, it is important to tend to passengers during the flight — which is done currently by flight attendants. While they are not pilots, they are still flight crew on board. If the goal is to totally remove humans from a commercial flight, their services have to be replaced with automated ones as well. This is especially important in small aircraft or flying taxi scenarios.

Current level of automation

Today’s commercial aircraft are heavily automated. These aircraft are no longer flown by directly controlling flight surfaces with a yoke or joystick or rudder through push-rod or pulley systems, buy employ ‘fly-by-wire’ systems, where the pilot provides commands to the aircraft, such as “roll to the right 5º”, and the aircraft executes on it’s own accord, doing whatever is necessary to achieve the desired outcome.

On top of flight controls, today’s aircraft also have extended autopilot functionality. Most commercial flights today start with downloading the flight route into the flight management system of the aircraft, pilot doing the taxi & take off roll, and after gears up, he switches to the autopilot, which flies his route to the destination. There, he either does a manual or automated landing, depending on weather, corporate policies and his personal preference. The lower the visibility, the more probable it’s an automated landing.

Today’s autopilots need a human pilot to supervise them. In fact, the autopilot has to be monitored at all times by a human pilot — one can’t leave the cockpit while the aircraft is on autopilot. Moreover, the route flown is seldom the one initially planned, but changes throughout the flight, due to weather, traffic routing or other reasons. Often times the pilot gets a ‘shortcut’, a clearance to fly more directly to his destination than in the original flight plan, which follows historic airways where Free Route Airspace has not yet been introduced. It is the human pilot who has to update the autopilot with the updated route, or enter a specific heading, a new altitude, an approach procedure, etc.

There is no automation today available for ground taxi before takeoff and after landing. Neither is there automation available for communicating with Air Traffic Control and updating the flight route based on ATC clearances.

From 5 people in the cockpit to 2, then 1

In the heyday of aviation, an aircraft cockpit might have included up to 5 people: 2 pilots (captain & first officer), a flight engineer, a navigator and a radio operator. They might have even had a second or third officer (more pilots). The pilot would for example increase thrust by issuing a verbal command to the flight engineer, who would then use a number of engine control levers and whatnot to increase thrust. The pilot would tune into a new communication frequency or radio navigation frequency by issuing a verbal command to the radio operator, who would use his deep knowledge of radio dialing to fine tune the frequencies, verify the radio navigation aid morse code ID, and then report back that the navaid has been tuned. Usually he would be the one to speak on the radio with Air Traffic Control or other aircraft. The navigator would keep track of aircraft position, update routes and so forth.

Today’s commercial airliners have 2 crew members in the cockpit, a captain and a first officer. Both pilots, they utilize the built in instruments and equipment to do all their work. All the work that in the past has been delegated to other human crew members are now handled by the built in systems of the aircraft.

The next step in this progression is to have a single pilot on board, with pilot unions naturally opposing the idea, as they have opposed any previous reduction in the past, from 5 to 2. One of their arguments is the need for redundancy, or the perils of a single point of failure if there is only a single pilot on board. Proponents of the single pilot cockpit aim to solve this issue by virtue of an “optionally piloted vehicle”, which is an aircraft that can be piloted both by a pilot on board, and through remote control — a pilot situated on the ground, having control over the flying aircraft. In such a scenario, the “first officer”, or backup pilot, is the one on the ground. The cost saving is obvious: one person less in the cockpit, a ground based remote pilot can monitor multiple aircraft at the same time, and be only called up in case of need, which will be the exception and not the rule.

There is still ultimately a human in control in the case of the single pilot or optionally piloted vehicle, be him in the aircraft or on the ground via remote control. Totally removing this human is a significant challenge.

One to Zero

The partial building blocks described above are good stepping stones to achieve a pilotless cockpit, but fundamentally more is required.

One crucial aspect of being a pilot in command is decision making in critical phases of flight. One of the first decisions is before the flight even starts: is the flight a “go”? Are all conditions, aircraft, weather, fuel, mass & center of gravity, etc., all favorable for a successful flight? And while some of this can easily be formalized, some of the decision making requires careful consideration and experience, especially edge cases that have no formalized answers and can only be resolved through experience and good judgement.

Another thing that is missing is a semantic understanding of the environment the aircraft is operating in, which can be the basis of decisions to be made, in lieu of the pilot. The ‘environment’ here is quite broad, and includes the state of the aircraft, the space / scenery around the aircraft, other flying objects, traffic routing, state of the aircraft and so forth.

Let’s take a look at the various aspects of situational awareness that is needed.

Position & the surrounding environment

A very important aspect of flying is to not hit anything at high speed — be that birds, other aircraft, or the ground. To ensure this doesn’t happen, one needs to be aware of his own position, and the surrounding environment that he’s in — with these two going hand in hand. A loss of such an understanding results in “Controlled Flight Into Terrain” accidents, which is the second highest fatal accident category after loss of control in flight.

Aircraft positioning is an established technology, mostly based on radio-navigation devices dating back to the first half of the 20th century. Current aircraft employ what is called Performance Based Navigation (PBN), which is a fusion of a set of technologies resulting in a real-time position of the aircraft with some specified level of precision. The technologies used is a fusion of radio navigation technologies like VORs and global navigation systems like GPS, GLONASS, Galileo et al, which can be augmented for high precision.

For various phases of flight, the required navigational (position) precision is different, which makes sense. Flights are coordinated as such that separation between aircraft is significantly greater than the position error that might come from the navigation system. En route the separation is greater and the precision requirement is lower. Close to ground, for example on approach & landing, separation is smaller and the required precision is higher.

Position in the form of a geo-coordinate is only one side of the coin. To avoid hitting things, one needs to know where they are. For the ‘ground’, traditionally this is achieved by having a navigation database loaded into the aircraft before flight, including a global elevation model (e.g. shape of the Earth), an obstacle database (high structures poking upwards) and aviation specific things of importance, like airspaces, navigation points, airways, and most importantly, runways on which one can land. While these databases are very detailed, inevitably they reflect a past snapshot of reality— and cannot reflect temporal changes in the environment.

Despite the limitations of the current approach, a lot can be achieved using them. All current autopilots operate with the above set of technologies, flying tens of thousands of commercial flights each day, a lot of them using existing autoland technology to land at large commercial airports. While the current ILS Cat III autoland solutions require expensive and complex ground equipment and procedures to operate, new, PBN / GPS-based autoland capabilities are entering the market as soon as 2020, which don’t require special equipment at the airport.

Another important factor is not to hit anything else that is in the air. Currently this problem is mostly solved for cooperating traffic — other aircraft that are “playing along” in terms of where everyone is in the air. Either by following Air Traffic Control guidance, or by mutual “seek & avoid” approaches, and preferably having active technology on board such as ADS-B or FLARM that share the aircraft’s position with others in the vicinity.

AI to the rescue

PBN based positioning and navigation databases are a good start, but only allow to avoid obstacles based on navigation databases that are a snapshot of the past, and offer to avoid flying objects that are cooperative and each of them utilize some nature of active technology. We all know that reality is more immediate than that, and that not everyone is cooperative — for example, birds aren’t born with ADS-B equipment embedded in them, the drone safety landscape is being formed as we speak, and there can be actors in bad faith out there too. Not to speak of animal incursions on a runway.

The concept of Performance Based Navigation is based on improving navigational precision through fusing position measurements from multiple sources into a more stable, redundant and precise outcome. A vision-based landing system, identifying the runway using camera imagery, can also add to the navigational precision of an aircraft when landing. For a vertical landing on a heliport or vertiport, visual recognition might be even simpler, with visual markings on the landing spot easily recognized by computer vision algorithms when looking straight down from the aircraft.

This concept can be generalized, with an optical 3D mapping approach through a 3D Simultaneous Localization and Mapping (SLAM) or similar algorithm to create a 3D map of the aircraft’s surrounding and concurrently determine the aircraft’s position in this 3D map. While some 3D SLAM technologies make use of extensive post-processing of imagery to create high resolution 3D models of an area, for navigation purposes one needs real time results but less 3D detail.

Ultimately these approaches will provide yet another form of input into the existing navigation systems of aircraft, with the advantage of having significantly different degradation characteristics than existing systems, thus adding resilience and redundancy.

Wait, what’s that?

Having a 3D map of the environment certainly helps, but is not sufficient — the system also needs to have an understanding of what is around it. After all, the 3D mapped object might be a cloud, which has very different implications that if it was a mountain. Airborne objects, due to their movement and size, will be missed by a 3D map — or ground objects that move too, due to their transient nature. When determining where to land, one needs to understand the type of surfaces on the ground, and also be sure if they are free of incursions or contamination. The vision based landing demonstration shown above is an example of such semantic understanding, albeit very specialized.

Deep learning can be used to classify the terrain according to flight mission needs — for example identify possible landing sites, or if there are incursions on a designated landing site. A similar approach can be used to identify flying objects and detect birds, drones, aircraft on a solely visual basis for purposes of avoiding or hunting them down.

Flight coordination

With the expectation that autonomous flying vehicles will increase the number of aircraft in the sky, and these will be densely concentrated around metropolitan areas, existing centralized air traffic control might not be viable to coordinate these flights. At the same time, leaving them in a traditional uncontrolled airspace environment might be counter productive too.

One approach to overcome this situation may be a swarm type, decentralized self-coordination approach, where all vehicles cooperate to devise the best possible overall routing for all airborne vehicles. Data-driven multi-agent systems are already evaluated in autonomous maritime contexts for a similar purpose. Such an approach may be facilitated by vehicle-to-vehicle communications similar to the automotive V2X systems, especially since existing airborne vehicle position reporting systems like ADS-B don’t scale — they can’t handle large number of aircraft in a small area.

Nothing is ever perfect — and doesn’t need to be

None of the systems, existing or future, are ever 100% perfect. Yet, we trust our lives to them, and they continue to earn our trust over extended periods of time. But how can imperfect systems work so reliably?

One way this is modeled is called the Swiss Cheese Model. Imagine a stack of cheese slices with holes in them, stacked layer by layer. We know that each layer has holes, but still, when looking at the full stack, it’s opaque. We’d need to penetrate several layers to fall through — even is at some layers we go through an existing hole, other layers around it will have to be penetrated through failure.

All-weather operation is key

In order to rely on flying taxi services, especially in time-critical use cases such as catching a flight at a major airport, these services have to work reliably and always. They need to work in day and night, they need to work in nice and bad weather.

All-weather, reliable operation is a key enabler of commercial air transport.

The first goal of current flying taxi aircraft, and also AI based autonomy systems is to fly in good weather during the day. This is a reasonable first step on the roadmap for any new aviation technology, but is only the first step. In order for people to rely on flying taxis on a regular basis, these aircraft have to operate — always.

The advancement in avionics in the past decades was mostly focused on removing the necessity for human pilots to visually identify their surroundings, and have instruments do the same, also working at night, in low or zero visibility, with longer range, higher precision and higher reliability. Re-introducing visual recognition of the environment, albeit with an AI, provides good redundancy, but brings back limitations related to reduced visibility, even if improved to some extent by visual enhancement technology.

The conclusion will be a fusion of various forms of sensor inputs, with different operational characteristics, resulting in a redundant and overall reliable multi-faceted approach. AI will not only be utilized at the sensor level, but also when fusing data from different sensors and during decision making.

Regulatory context

It is important to understand the regulatory context under which commercial flight takes place. Obviously this is a very tightly controlled area of flight, and for good reason. Both the US Federal Aviation Administration (FAA) and the European Union Safety Administration Agency (EASA) are in active discussion with industry players to come up with new regulations that facilitate fast, efficient and at the same time safe operation for autonomous aircraft, and at the same time starting to regulate a new aircraft category: eVTOL aircraft, more commonly known as flying taxis.

When developing autonomy features, the regulatory frameworks of aircraft types is of paramount importance. For example, for eVTOLs, EASA published a Special Condition for small-category VTOL aircraft, which includes provisions for both leisure flying (“Category Basic”) and flying over densely populated areas or performing commercial air transport (“Category Enhanced”) — flying taxis being the latter. One important difference is that for a “Basic” aircraft it is permissible to have single points of failure such that might only be resolved with an emergency landing out on the field, while for a “Enhanced” aircraft, no such events are allowed — even upon critical failure, the aircraft has to maintain controlled flight, reach and land at a designated landing area. This implies that for “Basic” aircraft, it might make sense to have an autonomous emergency landing capability that finds a nearby suitable landing spot and lands the aircraft, while for a “Enhanced” flying taxi, such a capability is not applicable, as it will have to be able to fly to a designated landing site anyway.

It is exciting to see new regulations being created for drastically new technologies and new types of aircraft at the same time, while our understanding of past regulatory practices is improving as well. There is an industry momentum to allow for a more diverse set of technologies to enter the market easier than before, arguing that even if a technology is not perfect, it is very much useful, makes pilots lives easier and saves lives too. Past examples include non-certified tablet based navigation devices, or the FLARM anti-collision system — initially both non-certified, but both being extremely useful. On the backbone of this argument, some manufacturers argue that when certifying aircraft, certain non-required safety features should give them a ‘certification credit’, and relax other aspects of certification. This is also important for vendors of such safety systems, as it gives aircraft manufacturers a clear incentive to incorporate their products, by saving on certification burdens due to the credit received for the additional safety feature that is incorporated.

Certifying AI

Certifying solutions incorporating Artificial Intelligence is of a peculiar challenge, due to the general approach of aviation certification, with a very large focus on determinism. For such systems to be certified, new approaches to certification needs to be adopted, with industry players and regulatory authorities working hand in hand. Ultimately a new frameworks needs to be created that can be used to show that these systems are indeed safe and perform as per expectations, even when they are not deterministic in the traditional sense.

Additional certification obstacles

A peculiar challenge in removing all flight crew from an aircraft is that some of the requirements on pilots is very human centered, albeit not strictly aviation related. For example, on a commercial flight, flight crew has to make sure that the passengers who board the aircraft are not intoxicated. How would a flight autonomy system do that? And why would it, in the first place? To allow for a totally crew-less commercial flight to happen, soft aspects some certifications need to be updated too.

Industry landscape

Flight autonomy is a very exciting landscape now. Industry veterans such as Garmin are pursuing this topic as a natural extension to their avionics offerings. Some eVTOL / flying taxi manufacturers like Volocopter and Lillium are building internal autonomy teams. Airbus’ Vahana flying taxi project is also building it’s internal autonomy team, which is a somewhat atypical move for such an OEM. Startups dedicated to this topic like Daedalean AI are aiming to be the go-to place for autonomy solutions. Drone pioneer Auterion is gradually incorporating autonomy features that can be scaled up in due time. Aurora Flight Sciences, now part of Boeing, is building an eVTOL with autonomy features and also working on optionally piloted vehicles, as does Autonodyne, which is associated with avionics veteran Avidyne.

Flying taxis will be operated mostly as ride-sharing services. Uber is pushing their own service through Uber Elevate, and is a very active participant and accelerator in this ecosystem. Some eVTOL companies plan to offer their own flying taxi service, covering the complete service experience.

How will this play out?

We’ll be seeing a gradual reduction of reliance on human pilots. Already now, autopilots supervised by pilots perform many aspects of commercial flights. Instead of two, we’ll only see one human pilot on board, with a remote backup in standby. Automated systems will be introduced to handle some emergency situations related to pilot incapacitation, which results in a sudden unavailability of a human pilot on board — turning a zero chance of survival into something more than zero. Eventually we’ll see flights that, after “doors closed”, will be able to perform a complete flight without a human pilot on board, with a remote pilot backup in standby. Most probably this will come earlier in a flying taxi context, due to the reduced complexity of the complete process. Humans will be still involved, as ground crew, embarkation and disembarkation of passengers, and remote safety pilots, each overseeing a dozen or more aircraft on average. Some predict that for some time, there would be separate airspaces for autonomous and human-piloted aircraft, with autonomous aircraft self-coordinating each other like a swarm.

This huge shift requires close cooperation between industry players and regulatory authorities, which is already happening. New technologies such as AI that will have to be certified are yet missing certification frameworks, which need to be developed. Ultimately these technologies will be incorporated into existing avionics and flight control frameworks, and will contribute to the current level of precision and redundancy available, while enabling automation of new use cases that could not be achieved until now.

Ákos Maróy is an aviation innovation expert, commercial pilot, the founder or Aero Glass - an award winning AR startup for pilots, and has been the member of the Daedalean AI team until recently.