## Abstract

Quantum key distribution (QKD) employed orbital angular momentum (OAM) for high-dimensional encoding enhances the system security and information capacity between two communication parties. However, such advantages are significantly degraded because of the fragility of OAM states in atmospheric turbulence. Unlike previous researches, we first investigate the performance degradation of OAM-based QKD by infinitely long phase screen (ILPS), which offers a feasible way to study how adaptive optics (AO) dynamically corrects the turbulence-induced aberrations in real time. Secondly, considering the failure of AO while encountering phase cuts, we evaluate the quality enhancement of OAM-based QKD under a moderate turbulence strength by AO after implementing the wrapped cuts elimination. Finally, we simulate that, with more realistic considerations; real-time AO can still mitigate the impact of atmospheric turbulence on OAM-based QKD even in the large wind velocity regime.

© 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

## 1. Introduction

Quantum key distribution (QKD) [1–6] provides unconditionally secure random numbers [7–12] between two authenticated distant parties (Alice and Bob) based on the no-cloning principle of quantum physics [13,14]. In conventional QKD schemes, photons carrying spin angular momentum are usually used to encode information [15–20]. However, the information capacity of this binary system is limited to $1$ bit per photon, making it more vulnerable to eavesdroppers’ attacks. Unlike spin angular momentum, the orbital angular momentum (OAM) [21] of light, associated with $l$ intertwined helical phase $e^{il\phi }$ of each photon in a beam, where $l$ takes integer values and is unbounded, offers an alternative way to encode information in an infinitely large Hilbert space and improve the information capacity of QKD system to more than $1$ bit per photon [22]. Moreover, a higher-order system possesses a higher security threshold [23–25], such as, for the BB84 protocol in dimension $5$, this threshold increases to the limit of $21\%$ compared to $11\%$ for dimension $2$.

Over the past decade, most previous experiments, including indoors [26–35] and outdoors [36–42], have demonstrated such encoding schemes are feasible up to dimension $7$ [30] and the range as far as $340m$ [38]. Nevertheless, the presence of atmospheric turbulence remains the greatest challenge to effectively implement an OAM-based QKD experiment in free space. Random air refractive index fluctuations caused by atmospheric turbulence disrupt the phase alignment of original transmitted optical fields, split the optical vortex into several individual vortices and create the photonic OAM pairs during the propagation [43–45], all combinations of which result in mode scrambling at the receiver [46–53], making the high dimensional encoding lose its unique advantages.

To mitigate the adverse effects of atmospheric turbulence, the common compensation strategy is to use adaptive optics (AO) [38,54–58]. Recently, many numerical experiments have been performed using AO correction in quantum levels [59–61]. However, all these scenarios are simulated by random phase screens, which is only valid if turbulence satisfies Taylor’s hypothesis [62] (i.e., when the laser pulse width is significantly narrower than the time scale of random refractive index fluctuations, atmospheric turbulence is assumed to be stationary during this tiny time fraction). To overcome this barrier, we employ the infinitely long phase screen (ILPS) [63] method to dynamically simulate the evolution of the average quantum bit error rate and the secret key rate of OAM-based QKD.

With the implementation of atmospheric propagation, the branch points [64] caused by intensity modulation will occur in the phase distribution at the receiver, broadening the OAM spectrum distribution [45,65] and leading to a reduced performance of OAM-based QKD. These phase discontinuities complicate the phase correction in an AO system. Considering the inability of traditional AO while encountering phase cuts [66–68], we evaluate the performance enhancement of OAM-based QKD by AO after implementing the wrapped cuts elimination. We also demonstrate that such localized manipulation is physically possible and brings a significant improvement compared to the previously poor correction.

It should be noted that this approach brings the drawback that it reduces the response rate of AO system [54,69] and tends to affect the correction performance of them while turbulence is changing rapidly [70] . Concretely, the slope of beacon light measured by a Hartmann wavefront sensor [71] is not converted into the voltage signal of a deformable mirror directly but is used to reconstruct the phase of beacon light. Then, the reconstructed phase is repaired by phase unwrapping algorithm and finally converted into the voltage signal of a deformable mirror. To examine the feasibility of our scheme, we re-evaluate the performance enhancement achieved by AO and conclude that this scheme is still able to mitigate the impact of atmospheric turbulence on OAM-based QKD, even in the large wind (we use the word "wind" in general referring to the crosswind) velocity regime.

This paper is structured as follows. Sec. 2 briefly introduces the two mutually unbiased bases (MUBs) used in OAM-based QKD and presents the principle of OAM-based QKD that integrates the effects of atmospheric turbulence. The main methods used in this paper, including modified ILPS, perturbed phase correction, and wrapped cuts elimination, are detailed in Sec. 3. In Sec. 4, we first investigate the undesirable impact of atmospheric turbulence on a single OAM state; secondly, we evaluate the correction effect of our enhanced AO to compensate the turbulence-induced aberrations for OAM-based QKD under arbitrary turbulence strengths and correction orders; finally, we consider the impact of realistic noise contributions on the performance of OAM-based QKD. At last, Sec. 5 discusses some deficiencies of our numerical methods and concludes this paper.

## 2. Physical model

#### 2.1 Photonic source

A photon’s degrees of freedom, such as time-energy, OAM, and position momentum, are typically physical sources for implementing high-dimensional encoding. For an OAM-based protocol, we study the two MUBs that consist of a group of successive OAM photon states and its complementary Fourier conjugate angular basis (i.e., so-called angular position (ANG) state) $\left \vert j\right \rangle =1/\sqrt {d}\sum _{l=-L}^{L}\left \vert l\right \rangle \exp \left ( -i2\pi jl/\left ( 2L+1\right ) \right )$, where $\left \vert l\right \rangle$ represents a single-photon state of a Laguerre-Gaussian (LG) mode [[72––––77]] $LG_{0,l}\left ( r,\phi ,0\right )$, $d=2L+1$ is the dimension of the encoding subspace and $L$ is the maximum azimuthal index used in this protocol. Without loss of generality, the LG modes at output plane $z$ can be described in cylindrical coordinates by

#### 2.2 Principle of OAM-based QKD

In the weak scintillation regime, the distortion of a photonic OAM state $\left \vert l_{0}\right \rangle$ propagated through atmospheric turbulence can be considered as a pure phase perturbation [46,47]. To investigate the influence of turbulence on OAM-based QKD under arbitrary scintillation conditions, the split-step method [82] is used to simulate the procedure of atmospheric propagation instead of the pure phase perturbation approximation. As shown in Fig. 1(a), multiple layers of ILPS are arranged consecutively to form a phase-screen array for simulating the propagation of two MUBs across the turbulence. For convenience of presentation, the unitary operator $\widehat {U}_{turb}^{\left ( i\right ) }$ is used to represent the $i$-th realization of a photonic state traveling through turbulence instead of the extended Huygens-Fresnel integral [60,61,84]. Hence, after undergoing the unitary transformation, the received state becomes a superposition of several LG modes, which can be described using ket-bra notation as follows [60,61]

Based on the principle of a single OAM state propagation, we transfer our attention to a concrete QKD scheme. Since Alice randomly chooses her photonic OAM states from two MUBs, we only concern the spectral broadening of initially transmitted modes within the subspace of primary encoding basis and regard the mode scrambling outside the subspace as the atmospheric losses during the simulation. To get the measured matrix of OAM-based QKD, we normalize the crosstalk probability of each incident OAM state, which can be expressed in the coordinate representation as [73]

In order to explore the impact of atmospheric turbulence on the performance of OAM-based QKD, this simulation ignores the noises arising from the detector and experimental instruments, such as dark counts [29], afterpulsing effect [87], to name just a few. Hence, the bit error arises from the turbulent channel can be quantified by the average quantum bits error rate (QBER). For the given two MUBs, the average QBER in OAM basis is [88]

Likewise, the average QBER in ANG basis $Q_{ANG}^{\left ( i\right ) }$ can also be acquired from the same procedure. By averaging the QBER over two MUBs, the total errors caused by atmospheric turbulence are described as

Finally, we evaluate the information capacity that is securely exchanged between both parties before the postselection for OAM-based QKD, quantified by the minimum secret key rate, which can be calculated as [25,88]

## 3. Numerical methods

#### 3.1 Modified infinitely long phase screen

The simplest method to simulate atmospheric turbulence is to use the random phase screen [89], together with a low-spatial frequency compensation [90]. However, the random phase screen is established on the fact that laser pulse width is significantly narrower than the time scale of random refractive index fluctuations [62]. Therefore, the dynamical process of incident light propagating through atmospheric turbulence and how AO implements dynamical correction to the perturbation cannot be investigated within this method (e.g., the execution rate of AO system is greater than the variation time of turbulence). Moreover, the static assumption will lose its usefulness when studying the influence of wind velocity on an AO-aided OAM-based QKD.

The ILPS method gives a feasible solution to overcome this barrier. Without loss of generality, we summarize and divide the procedure of ILPS’s execution into two steps as follows [63]: Firstly, we generate an initially random phase screen and calculate the pixel number (i.e., rows or columns) to be moved (we call this as the need-to-be-moved pixel number) in each iteration according to the wind velocity $v$, wind direction $\theta$ and iteration time (i.e., time interval between two iterations). Secondly, for the $i$-th realization of turbulence and $j$-th need-to-be-moved pixel, we calculate the new need-to-be-added row or column $X$ from the partial data $Z$ of last-generated phase screen, add it into the last-generated one and discard the excess part of the oversized phase screen (a more straightforward illustration is presented in Fig. 1(b)). The relationship between $X$ and $Z$ is given by [63]

where $Z$ represents the last $N_{col}$ rows/columns of last-generated phase screen, $\beta$ is a Gaussian random vector with zero mean and its covariance equal to unity. The matrix $A$ and $B$ (see more details for the derivation of $A$ and $B$ and their dimension informations in Ref. [63] ) can be calculated from the covariance of the $X$ and $Z$ vectors, including the matrices $\left \langle ZZ^{T}\right \rangle$, $\left \langle XZ^{T}\right \rangle$, $\left \langle ZX^{T}\right \rangle$ and $\left \langle XX^{T}\right \rangle$. Concretly, all of covariance matrices can be acquired from constructing the distance matrix $r_{m,n}\equiv R\left ( X_{m},Z_{n}\right )$ and acting them on the phase covariance function $C_{\varphi }\left ( r\right )$, where $X_{m}$, $Z_{n}$ and $R\left ( \cdot \right )$ denote the $m$-th element in $X$, the $n$-th element in $Z$ and the distance function respectively. If the turbulence spectrum follows the von-Karman rules, $C_{\varphi }\left ( r\right )$ can be expressed as [91]It is worth noting that the conventional ILPS (we use the word "conventional" in general referring to the ILPS mentioned in Ref. [63] ) can only simulate the effects of wind velocity changes in both horizontal and vertical directions, which significantly limits the utilizing of this algorithm. For this reason, we modify the conventional ILPS by decomposing the wind velocity into horizontal and vertical directions. For simplicity, we now detail our modifications into two parts: On the one hand, our algorithm first detects whether a new column needs to be added in the horizontal direction, and if so, subtracts one pixel number from the horizontally need-to-be-moved pixel number after each motion (if we assume the need-to-be-moved pixel number equals to $N_{move}$, then the horizontally and vertically need-to-be-moved pixel number equal to $N_{move}\sin \theta$ and $N_{move}\cos \theta$ respectively), then, executes the same procedure in the vertical direction and repeats the two steps until no motion demand is required in either direction. On the other hand, after undergoing above manipulation, the residual pixel number of two directions (i.e., the difference between the horizontally/vertically need-to-be-moved pixel number and the physically moved pixel number in two directions), which is commonly less than one pixel number, is added to the horizontally and vertically need-to-be-moved pixel number for the next iteration [92] (see two examples of modified ILPS in Fig. 2).

#### 3.2 Adaptive optics system

### 3.2.1 Perturbed phase correction

A well-developed AO system comprises three primary assignments: wavefront measurement, reconstruction, and beam stabilization. The slope magnitude of beacon light is measured by a Hartmann wavefront sensor, and subsequently used to invert the voltage signal, which is used to generate the phase distribution of a deformable mirror, through a reconstructor. For a closed-loop AO system, the estimated phase is also fed back to compensate the slope magnitude [93], the combinations of which can continuously improve the phase reconstruction accuracy.

Since the phase generated by AO system is only an approximation of the perturbed phase of beacon light, the correction capability of entire system can be commonly measured by the number of coefficient that is calculated from the spectral decomposition of beacon light using Zernike polynomial [57,58]. (e.g., for an ideal AO system, the correction order can reach infinity because the perturbed phase of beacon light can be expanded to a superposition of an infinite number of Zernike polynomial with different orders, which is usually considered as an upper boundary for the performance enhancement of AO). For a realistic AO system with $N$-th order correction capability, the phase estimated by a deformable mirror can be expressed as

Finally, we emphasize that, in our AO-aided OAM-based QKD scheme, the signal part of incident modes consists of photonic OAM states and their Fourier conjugate angular states (i.e., ANG states). Besides, we employ the platform beam as a probe state to detect atmospheric turbulence [94]. Our simulations ensure that two beams are emitted simultaneously and propagated coaxially to provide a better correction for the signal part.

### 3.2.2 Wrapped cuts elimination

A realistic AO system has been proven to be ineffective at compensating for branch points that occur at places of zero amplitude in an optical field [95]. These unavoidable phase cuts [64] degrade the performance of OAM-based QKD due to the inability of a continuous surface deformable mirror to reconstruct the discontinuous phase [66–68]. Besides, some previously correlated researches have demonstrated that branch points begin to appear when the Rytov variance exceeds $0.1$ [96,97]. Reassuringly, this adverse effect is not completely insurmountable. Inspired by Refs. [98], we employ the phase unwrapping algorithm proposed in Ref. [99] to enhance the performance of AO correction in this paper.

The main idea of our enhanced AO scheme is that we reconstruct the perturbed phase using the real-time slope information of beacon light and then unwrap the discontinuous phase by an effective and implementable algorithm; subsequently, the unwrapped perturbed phase is used to generate the voltage signal of a deformable mirror. Generally, a standard unwrapping operation on the perturbed phase $\varphi ^{\left ( i\right ) }\left ( r,\phi ,z\right )$ can be accomplished by adding an integer multiple of $2\pi$ at each pixel of wrapped phase, which is mathematically expressed as [100]

## 4. Results

#### 4.1 Single OAM state propagation

To quantify the undesirable impact of atmospheric turbulence on OAM-based QKD and what effects can be obtained through partial correction by AO, the evolution of a single OAM state that is perturbed while traveling across the turbulent channel and repaired by a specific-order AO are investigated. In Fig. 3, we plot the phase distributions of a state $l_{0}=3$ for a single realization of turbulence with no correction (Fig. 3(a) and 3(d)), realistic correction [101] (Fig. 3(b) and 3(e)) and ideal correction (Fig. 3(c) and 3(f)), where the upper and lower three diagrams represent the results realized in the weak and strong scintillation regime, respectively (see more details about the parameter settings in Ref. [102]).

Figure 3(a) shows that the phase distribution of OAM state is slightly distorted after propagation. However, this circumstance becomes more severe in the strong scintillation regime (Fig. 3(d)), which will lead to a completely disrupted structure. Notably, we observe in Fig. 3(a) that the initial vortex splits into three individual vortices, accompanied by the vortex-antivortex pairs regeneration at places of central point of initial one. Besides, turbulence also leads to a longer discontinuous cut in the perturbed phase distribution (For more detailed explanations, we refer the reader to Refs. [64,65]). To eliminate the phase discontinuities, a realistic AO is employed to repair the perturbed phase distribution, as shown in Fig. 3(b) and 3(e) ((b) $30$-order realization (e) $50$-order realization). It is highlighted that, in the weak scintillation regime, the longest phase cut that occurs in the perturbed phase is completely eliminated, making the unwrapped corrected phase is more approach to the unperturbed one (see Fig. 3(b)). On the contrary, in the strong scintillation regime, the corrected phase remains almost unchanged compared to the perturbed one (see Fig. 3(e)). Fortunately, such circumstance is significantly alleviated in ideal correction (see Fig. 3(f)).

The poor correction is likely because the length of branch cuts becomes a crucial factor in affecting AO correction effectiveness. In other words, in the weak scintillation regime, only a few branch cuts are generated such that these can be ignored when we implement realistic AO correction (see more discussions in Ref. [45]). However, although we eliminate all wrapped cuts in the strong scintillation regime, the accumulation of branch cuts will destroy the phase distribution of the beacon light eventually.

In order to evaluate the correction effect of AO intuitively, the crosstalk probability distributions (including the radial-mode scrambling) averaged over $500$ realizations of turbulence are presented in Fig. 4. We observe that, in the weak scintillation regime, our enhanced AO scheme effectively restores the power into the initial OAM state even in the presence of phase cuts. Conversely, in the strong scintillation regime, we note that our enhanced AO loses its advantage to mitigate turbulence-induced crosstalk with the increase of branch points. Furthermore, we also find the well-known result [61,103,104] that the spectral broadening peaks around the incident state. In the strong scintillation regime, the crosstalk distribution has two peaks around $l_{0}=3$ and $l_{0}=-3$ (For more discussions about the bimodal distribution, we refer the reader to Ref. [61]), which results in a smaller average OAM of the received state compared to that of the incident one [43].

#### 4.2 Performance of OAM-based QKD across turbulence

Based on the above analysis for the propagation characteristics of a single OAM state, we now employ our enhanced AO to improve the quality of OAM-based QKD (without (a) and with (b) increasing the mode spacing in encoding subspace). Figure 5 illustrates the variation curves of average QBER without, with realistic and with ideal AO correction recorded during half an hour for 5-dimensional OAM-based QKD system. we set the correction capability of realistic AO to $30$-order and $C_{n}^{2}$ to $2.2\times 10^{-15}m^{-2/3}$ during this simulation, where $C_{n}^{2}$ is the refractive index structure constant (see more detailed parameter settings in Ref. [102] and reasons in Ref. [105]).

As illustrated in Fig. 5(a), we find that, without increasing mode spacing (i.e., using $\left \vert -2\right \rangle$, $\left \vert -1\right \rangle$, $\left \vert 0\right \rangle$, $\left \vert 1\right \rangle$ and $\left \vert 2\right \rangle$ for encoding), the average QBER can be mitigated from $41.9\%$ with a standard deviation of $6.14\%$ to $11.1\%$ with a standard deviation of $2.93\%$ through the realistic AO correction. Notably, for the ideal circumstance, the average QBER is reduced by $39.63\%$. With the help of mode spacing increasing (i.e., using $\left \vert -4\right \rangle$, $\left \vert -2\right \rangle$, $\left \vert 0\right \rangle$, $\left \vert 2\right \rangle$ and $\left \vert 4\right \rangle$ for encoding, as shown in Fig. 5(b)), we can alleviate the average QBER from $41.9\%$ to $33.1\%$ without correction, $11.1\%$ to $6.9\%$ with realistic correction and $2.3\%$ to $0.9\%$ with ideal correction, which indicates that the combination of two strategies can lead to a significant reduction of average QBER caused by atmospheric turbulence and rebuild a secure channel between two communication parties. In Fig. 6, we also evaluate the measured crosstalk matrices of perturbed OAM basis under the same settings of Fig. 5 without and with AO correction respectively. More detailed discussions about the crosstalk matrices are similar to the analysis presented in Fig. 5.

However, will AO correction work so well in the real-world? Unfortunately, when we implement our enhanced AO-aided OAM-based QKD experiment, the instrument inevitably generates some intrinsic noises (such as dark counts [29], shot noise [73], afterpulsing effect [87], time delay effect [54,69,70], to name just a few); besides, the two MUBs will also encounter different degrees of loss due to the receiving aperture [73], atmospheric aerosols and dust particles [106] during the propagation (the last two factors are not considered in this contribution, see reasons in Ref. [107,108]). The accumulation of these realistic noises can remarkably lead to an increased average QBER and seriously degrade the performance of realistic AO system. Hence, we have to emphasize that the reason why AO correction can achieve the high-quality enhancement in the above simulation is that the above noise contributions are not considered, which will be mentioned in subsection 5.

#### 4.3 Under arbitrary turbulence strengths

In this section, we describe how the average QBER and the secret key rate of OAM-based QKD change with respect to the atmospheric coherence length $r_{0}$ under different dimensions of encoding in Fig. 7, where each column from left to right represents the results without (Fig. 7(a) and (d)), with realistic (Fig. 7(b) and (e)) and with ideal (Fig. 7(c) and (f)) AO correction respectively. The dashed vertical lines located at $r_{0}\approx 0.017m$ in all six plots correspond to the onset of weak scintillation. The atmospheric and optical parameter settings adopted in this subsection are listed as follows: $v=1m/s$, $\theta =\pi /2$, $z=1000m$, $w _{0}=0.03m$ and $\lambda =632nm$.

From the results presented in Fig. 7(a), we observe that the average QBER gradually decreases as the turbulence becomes weaker. For a higher dimensional OAM-based QKD, the average QBER is overall superior to the lower dimensional encoding in the strong scintillation regime, which is likely explained by the fact that the OAM eigenstates with a higher azimuthal index have a larger beam size so that they are more susceptible to atmospheric turbulence. In the weak scintillation regime, all these differences between different curves become smaller with decreasing turbulence strength. Besides, as illustrated in Fig. 7(d), we see that no positive key rate can be obtained by OAM-based QKD in all considered dimensions under moderate to strong turbulence strengths (i.e., $r_{0}$ ranges from $0.01m$ to $0.06m$), which causes both parties cannot build a secure communication link. Generally, we expect to improve the system security and information capacity of OAM-based QKD when we employ high dimensional encoding. However, we find in Fig. 7(d) that when $r_{0}\approx 0.12m$, a lower secret key rate will be obtained despite a higher dimensional encoding subspace is used (i.e., a more secure communication link guaranteed by a higher dimensional OAM-based QKD is destroyed by atmospheric turbulence). Fortunately, high dimensional OAM-based QKD will reveal its unique advantages as the turbulence strength gradually decreases. Therefore, it should be noted that, for a given turbulence strength, selecting an appropriate dimension for encoding is especially critical to acquire a better performance of OAM-based QKD (e.g., we choose $d=3$ for $r_{0}\approx 0.12m$ and $d=7$ for $r_{0}\approx 0.25m$).

The central and last columns in Fig. 7 illustrate the QKD’s performance achieved with realistic and ideal AO correction. Based on the comparison between Fig. 7(a) and (b), we see that, in the weak scintillation regime, the average QBER significantly decreases as the turbulence becomes weaker, which implies that our enhanced AO is effective for improving the quality of OAM-based QKD. Specifically, we notice that the realistic AO can achieve the best performance enhancement under moderate turbulence strengths (e.g., when $r_{0}=0.044m$ and $d=5$, the average QBER decreases from $44\%$ to $14\%$ and the secret key improves from no positive keys to $0.605$ bits/photon). On the other hand, we re-evaluate in Fig. 7(e) how the secret key rate changes with respect to $r_{0}$ under the realistic AO correction. Compared to Fig. 7(d), we observe that the realistic AO correction can lead to a positive key rate when the turbulence strength is moderate, which substantially improves the utilization of OAM-based QKD. Furthermore, with the help of ideal correction, we obtain an upper boundary (Fig. 7(c) and (f)) for the performance enhancement of OAM-based QKD, which leads to a positive key rate in all considered turbulence strengths (e.g., when $r_{0}=0.02m$ and $d=5$, the secret key rate improves from no positive keys to $1.04$ bits/photon).

We also enhance the performance of OAM-based QKD through increasing mode spacing in encoding subspace. Comparing the results obtained between the two strategies (Figs. 7 and 8), we note that, in the strong scintillation regime, when the mode spacing becomes $2$, the average QBER of OAM-based QKD counterintuitively increases compared to the successive encoding (see the comparison between Figs. 7(a) and 8(a), e.g., when $r_{0}=0.01m$, the average QBER without AO correction improves from $75.32\%$ to $80.32\%$ for $d=7$, $68.98\%$ to $73.95\%$ for $d=5$ and $56.49\%$ to $60.88\%$ for $d=3$, more explanations are presented in Ref. [38]). However, such performance degradation does not appear when we employ the ideal correction (see the comparison between Figs. 7(c) and 8(c)), which is a compromise result between the degradation and correction. Besides, in the weak scintillation regime, we conclude that increasing the mode spacing can partially improve the quality of OAM-based QKD, especially with the help of AO (e.g., when $r_{0}=0.15m$ and $d=5$, the secret key rate improves from $1.85$ bits/photon to $2.1$ bits/photon with realistic AO correction and from $2.04$ bits/photon to $2.27$ bits/photon with ideal AO correction).

#### 4.4 Under arbitrary correction orders

Since the correction capability of realistic AO system is commonly determined by its order, we evaluate how the performance of an AO-aided OAM-based QKD changes with respect to the correction order under different dimensions of encoding in Fig. 9. We set $C_{n}^{2}=1.01\times 10^{-14}m^{-2/3}$ during the simulation. It can be seen that the quality of OAM-based QKD improves significantly as the correction order of AO increases. However, such significant performance enhancement is only in the lower order correction range. The primary reason because the turbulence-induced aberrations is almost entirely concentrated in the low-frequency part. On the other hand, since we set the turbulence strength in the moderate to strong range, we notice that when the correction order is larger than $100$-order, the quality of OAM-based QKD still improves somewhat as the correction order increases. Generally, we anticipate that the quality of OAM-based QKD might not be significantly enhanced with the increase of the correction order of AO under the weak turbulence strengths. In Fig. 9(b) and 9(d), we also present the results of increasing mode spacing in encoding subspace. We observe that only a minor performance enhancement appears compared to the successive encoding.

Another feature can be seen in Fig. 9(c) and (d) is that when we increase the correction order of AO and mode spacing in encoding subspace, we can recover the advantage of high-dimensional encoding to some extent, but choosing $3$-dimensional OAM states for encoding is still the best choice under this turbulence strength.

#### 4.5 Realistic considerations

Now, we evaluate the performance enhancement achieved by our enhanced AO with more realistic considerations, including the effects of loss from a limited sized receiving aperture and time delay between wavefront measurement and correction.

### 4.5.1 Loss

The state-dependent loss [73,109] caused by the diffraction effects significantly degrades the performance of OAM-based QKD in long-distance propagation. In the two MUBs, an OAM eigenstate with a larger azimuthal number suffers from more significant loss and acquires more propagation phase [83] due to the limited size of receiving aperture, leading to a reduced quality of QKD even in the absence of turbulence. In order to evaluate the impact of state-dependent loss on the performance of OAM-based QKD, we re-examine how the average QBER and the secret key rate change with respect to $r_{0}$ under different sizes of receiving aperture. Here, we only consider the performance degradation in 5-dimensional OAM-based QKD. the other parameter settings adopted during the simulation is the same as that in subsection 3.

As illustrated in Fig. 10(a), we observe that when the channel is under strong turbulence strengths and without AO correction, reducing the receiving aperture can surprisingly mitigate the impact of atmospheric turbulence to some extent (e.g., when $r_{0}=0.034m$, the average QBER decreases from $53.4\%$ to $44.98\%$), which is somewhat counterintuitive. The main reason may be that, under these turbulence strengths, the smaller size of the aperture is similar to a low-pass filter [110], a large amount of turbulence-induced vortex-antivortex pairs may wander outside the receiving aperture and result in a decrease of average QBER [111,112]. Conversely, in the presence of AO correction, we notice that decreasing the size of receiving aperture will lead to a significant increase to average QBER (Fig. 10(b) and (c), e.g., when $r_{0}=0.01m$, the average QBER increases from $23.77\%$ to $35.93\%$ with ideal AO correction) and decrease to secret key rate (Fig. 10(e) and (f), e.g., when $r_{0}=0.31m$, the secret key rate decreases from $2.89$ bits/photon to $1.32$ bits/photon with ideal AO correction). Such an observation is likely because the aperture loss becomes the primary source of bit error for OAM-based QKD after implementing AO correction.

### 4.5.2 Time delay

In above simulation, we investigate the quality enhancement of OAM-based QKD based on the assumption that AO correction is implemented in real time. However, since our enhanced AO includes the procedure of reconstructing and unwrapping the phase of beacon light, the impact of time delay effect between wavefront measurement and correction on OAM-based QKD cannot be ignored, which can also significantly affect the performance of OAM-based QKD. It is well-known that real-time implementation can not be realized except the Greenwood frequency is small enough for the bandwidth of AO system [113]. In order to re-evaluate the impact of performance degradation caused by the time delay effect on OAM-based QKD, we set the bandwidth of AO as one-fifth of the sampling frequency of a Hartmann camera [114] (i.e., if we assume the camera sample frequency equals to $1KHz$, then $f_{AO}=200Hz$), and change the Greenwood frequency $f_{G}$ by adjusting the wind velocity through the following relationship [54]

In Fig. 11, we consider how the average QBER changes with respect to $r_{0}$ under different wind velocities. We observe that when the wind velocity is small (i.e., $v=1m/s$), the evolution of average QBER almost coincides with the results obtained without time delay. On the contrary, with the increase of wind velocity, the average QBER starts to deviate from the no time delay counterparts, leading to the performance degradation of OAM-based QKD (e.g., when $r_{0}=0.026m$, the average QBER increases from $28.34\%$ to $40.54\%$ for $v=5m/s$, from $28.34\%$ to $49.78\%$ for $v=10m/s$). Besides, we also notice that when the turbulence becomes weaker, the deviation together inclines to disappear. To explain above phenomena, Fig. 11 is horizontally divided into two parts [115]. We observe that when $v=5m/s$ and the Greenwood time (defined as $\tau _{G}\equiv 1/f_{G}$) passes this threshold, the bandwidth of AO is large enough to provide a comprehensive correction (see more discussions in Ref. [38]). However, these conclusions are not established for $v=10m/s$, which implies that, in the large wind velocity regime, the bandwidth of AO needs to be enhanced to satisfy the fast time-varying wavefront sampling and compensation. Finally, by comparing to the results obtained without AO correction, we see that AO remains able to mitigate the performance degradation even though it works in the large wind velocity environment.

## 5. Discussion and conclusion

In this paper, we employ the ILPS method to simulate the performance degradation of OAM-based QKD under arbitrary turbulence strengths. The main idea behind ILPS is to use the partial data from previous phase screen to generate the results of next one, which builds a connection between two generated phase screens. An advantage of this design is that the wind velocity of turbulence can be simulated by adjusting the amount of motion in each iteration, which provides a feasible way to simulate the dynamical process of incident light propagation and how AO dynamically corrects the turbulence-induced aberrations in real time. However, we have to highlight the shortcoming of ILPS, which uses the spatial correlation of phase screen between two iterations to replace the temporal correlation of turbulence variation (e.g., the variation of wind velocity increases the Greenwood frequency of turbulence [54,70], however, ILPS does not perform well in the face of turbulence variation in a windless environment), a topic needs to be addressed in the future.

Considering the inability of traditional AO while encountering phase cuts [66–68], we demonstrate the feasibility of AO that includes the wrapped cuts elimination, which significantly alleviates the impact of atmospheric turbulence on OAM-based QKD. We show that, in the weak scintillation regime, the employment of realistic AO for correcting the turbulence-induced aberrations can lead to a $30\%$ performance enhancement approximately. However, in the strong scintillation regime, we observe a rapid reduction of the performance of AO correction. Furthermore, we suggest that if we wish to obtain a high-quality performance enhancement, an advanced AO system is necessary for the realistic experiments [38].

Since the additional noise contributions (such as state-dependent loss, time delay effect of AO, to name just a few) degrade the high-quality enhancement achieved by AO, we re-evaluate the correction effect of AO and conclude that AO can still rebuild a secure communication channel and recover the high information capacity of OAM-based QKD, even in the increased loss and wind velocity environment. However, for the real-time correction consideration, we still need to enhance the bandwidth of AO when the wind velocity becomes larger.

Finally, it should be noted that although our enhanced AO includes the procedure for eliminating wrapped cuts, the branch cuts caused by turbulence-induced aberrations cannot be entirely negated by traditional AO techniques. Further research on minimizing the length of branch cuts that occur in the perturbed phase may prove fruitful.

## Funding

State Key Laboratory of Pulsed Power Laser Technology Supported by Open Research Fund of State Key Laboratory of Pulsed Power Laser Technology (2019ZR07); National Natural Science Foundation of China (11904369); Natural Science Foundation of Anhui Province (1908085QA37).

## Acknowledgments

The authors are very grateful to the reviewers for their valuable comments. Z. T. thanks Prof. Ruizhong Rao for his careful reading and insightful suggestions.

## Disclosures

The authors declare no conflicts of interest.

## Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

## References

**1. **C. H. Bennett and G. Brassard, “Quantum cryptography: public key distribution and coin tossing,” Proceedings of the IEEE International, Conference on Computers, Systems and Signal Processing pp. 175–179 (1984).

**2. **C. H. Bennett, “Quantum cryptography using any two nonorthogonal states,” Phys. Rev. Lett. **68**(21), 3121–3124 (1992). [CrossRef]

**3. **D. Bruß, “Optimal eavesdropping in quantum cryptography with six states,” Phys. Rev. Lett. **81**(14), 3018–3021 (1998). [CrossRef]

**4. **V. Scarani, A. Acín, G. Ribordy, and N. Gisin, “Quantum cryptography protocols robust against photon number splitting attacks for weak laser pulse implementations,” Phys. Rev. Lett. **92**(5), 057901 (2004). [CrossRef]

**5. **A. K. Ekert, “Quantum cryptography based on Bell’s theorem,” Phys. Rev. Lett. **67**(6), 661–663 (1991). [CrossRef]

**6. **C. H. Bennett, G. Bennett, and N. D. Mermin, “Quantum cryptography without Bell’s theorem,” Phys. Rev. Lett. **68**(5), 557–559 (1992). [CrossRef]

**7. **H. K. Lo and H. F. Chau, “Unconditional security of quantum key distribution over arbitrarily long distances,” Science **283**(5410), 2050–2056 (1999). [CrossRef]

**8. **P. Shor and J. Preskill, “Simple proof of security of the BB84 quantum key distribution protocol,” Phys. Rev. Lett. **85**(2), 441–444 (2000). [CrossRef]

**9. **B. Kraus, N. Gisin, and R. Renner, “Lower and upper bounds on the secret-key rate for quantum key distribution protocols using one-way classical communication,” Phys. Rev. Lett. **95**(8), 080501 (2005). [CrossRef]

**10. **R. Renner, N. Gisin, and B. Kraus, “Information-theoretic security proof for quantum-key-distribution protocols,” Phys. Rev. A **72**(1), 012332 (2005). [CrossRef]

**11. **V. Scarani, H. Bechmann-Pasquinucci, N. J. Cerf, M. Dušek, N. Lütkenhaus, and M. Peev, “The security of practical quantum key distribution,” Rev. Mod. Phys. **81**(3), 1301–1350 (2009). [CrossRef]

**12. **D. Gottesman, H. K. Lo, N. Lütkenhaus, and J. Preskill, “Security of quantum key distribution with imperfect devices,” Quantum Inf. Comput. **4**(5), 325–360 (2004). [CrossRef]

**13. **W. K. Wootters and W. H. Zurek, “A single quantum cannot be cloned,” Nature **299**(5886), 802–803 (1982). [CrossRef]

**14. **N. Gisin, G. Ribordy, W. Tittel, and H. Zbinden, “Quantum cryptography,” Rev. Mod. Phys. **74**(1), 145–195 (2002). [CrossRef]

**15. **C. Gobby, Z. L. Yuan, and A. J. Shields, “Quantum key distribution over 122 km of standard telecom fiber,” Appl. Phys. Lett. **84**(19), 3762–3764 (2004). [CrossRef]

**16. **A. Poppe, A. Fedrizzi, R. Ursin, H. R. Böhm, T. Lorü nser, O. Maurhardt, M. Peev, M. Suda, C. Kurtsiefer, H. Weinfurter, T. Jennewein, and A. Zeilinger, “Practical quantum key distribution with polarization entangled photons,” Opt. Express **12**(16), 3865–3871 (2004). [CrossRef]

**17. **T. Schmitt-Manderbach, H. Weier, M. Fürst, R. Ursin, F. Tiefenbacher, T. Scheidl, J. Perdigues, Z. Sodnik, C. Kurtsiefer, J. G. Rarity, A. Zeilinger, and H. Weinfurter, “Experimental demonstration of free-space decoy-state quantum key distribution over 144 km,” Phys. Rev. Lett. **98**(1), 010504 (2007). [CrossRef]

**18. **M. Sasaki, M. Fujiwara, H. Ishizuka, W. Klaus, K. Wakui, M. Takeoka, S. Miki, T. Yamashita, Z. Wang, A. Tanaka, K. Yoshino, Y. Nambu, S. Takahashi, A. Tajima, A. Tomita, T. Domeki, T. Hasegawa, Y. Sakai, H. Kobayashi, T. Asai, K. Shimizu, T. Tokura, T. Tsurumaru, M. Matsui, T. Honjo, K. Tamaki, H. Takesue, Y. Tokura, J. F. Dynes, A. R. Dixon, A. W. Sharpe, Z. L. Yuan, A. J. Shields, S. Uchikoga, M. Legré, S. Robyr, P. Trinkler, L. Monat, J. B. Page, G. Ribordy, A. Poppe, A. Allacher, O. Maurhart, T. Länger, M. Peev, and A. Zeilinger, “Field test of quantum key distribution in the Tokyo QKD Network,” Opt. Express **19**(11), 10387–10409 (2011). [CrossRef]

**19. **J. Yin, Y. Cao, Y.-H. Li, S.-K. Liao, L. Zhang, J.-G. Ren, W.-Q. Cai, W.-Y. Liu, B. Li, H. Dai, G.-B. Li, Q.-M. Lu, Y.-H. Gong, Y. Xu, S.-L. Li, F.-Z. Li, Y.-Y. Yin, Z.-Q. Jiang, M. Li, J.-J. Jia, G. Ren, D. He, Y.-L. Zhou, X.-X. Zhang, N. Wang, X. Chang, Z.-C. Zhu, N.-L. Liu, Y.-A. Chen, C.-Y. Lu, R. Shu, C.-Z. Peng, J.-Y. Wang, and J.-W. Pan, “Satellite-based entanglement distribution over 1200 kilometers,” Science **356**(6343), 1140–1144 (2017). [CrossRef]

**20. **S.-K. Liao, W.-Q. Cai, W.-Y. Liu, L. Zhang, Y. Li, J.-G. Ren, J. Yin, Q. Shen, Y. Cao, Z.-P. Li, F.-Z. Li, X.-W. Chen, L.-H. Sun, J.-J. Jia, J.-C. Wu, X.-J. Jiang, J.-F. Wang, Y.-M. Huang, Q. Wang, Y.-L. Zhou, L. Deng, T. Xi, L. Ma, T. Hu, Q. Zhang, Y.-A. Chen, N.-L. Liu, X.-B. Wang, Z.-C. Zhu, C.-Y. Lu, R. Shu, C.-Z. Peng, J.-Y. Wang, and J.-W. Pan, “Satellite-to-ground quantum key distribution,” Nature **549**(7670), 43–47 (2017). [CrossRef]

**21. **L. Allen, M. W. Beijersbergen, R. J. C. Spreeuw, and J. P. Woerdman, “Orbital angular momentum of light and the transformation of Laguerre-Gaussian laser modes,” Phys. Rev. A **45**(11), 8185–8189 (1992). [CrossRef]

**22. **H. Bechmann-Pasquinucci and W. Tittel, “Quantum cryptography using larger alphabets,” Phys. Rev. A **61**(6), 062308 (2000). [CrossRef]

**23. **H. Bechmann-Pasquinucci and A. Peres, “Quantum cryptography with 3-state systems,” Phys. Rev. Lett. **85**(15), 3313–3316 (2000). [CrossRef]

**24. **N. J. Cerf, M. Bourennane, A. Karlsson, and N. Gisin, “Security of Quantum Key Distribution Using *d*-Level Systems,” Phys. Rev. Lett. **88**(12), 127902 (2002). [CrossRef]

**25. **L. Sheridan and V. Scarani, “Security proof for quantum key distribution using qudit systems,” Phys. Rev. A **82**(3), 030301 (2012). [CrossRef]

**26. **S. Grölacher, T. Jennewein, A. Vaziri, G. Weihs, and A. Zeilinger, “Experimental quantum cryptography with qutrits,” New J. Phys. **8**(5), 75 (2006). [CrossRef]

**27. **M. Malik, M. O’Sullivan, B. Rodenburg, M. Mirhosseini, J. Leach, M. P. J. Lavery, M. J. Padgett, and R. W. Boyd, “Influence of atmospheric turbulence on optical communications using orbital angular momentum for encoding,” Opt. Express **20**(12), 13195–13200 (2012). [CrossRef]

**28. **V. D’Ambrosio, E. Nagali, S. P. Walborn, L. Aolita, S. Slussarenko, L. Marrucci, and F. Sciarrino, “Complete experimental toolbox for alignment-free quantum communication,” Nat. Commun. **3**(1), 961 (2012). [CrossRef]

**29. **M. Mafu, A. Dudley, S. Goyal, D. Giovannini, M. McLaren, M. J. Padgett, T. Konrad, F. Petruccione, N. Lükenhaus, and A. Forbes, “Higher-dimensional orbital-angular-momentum-based quantum key distribution with mutually unbiased bases,” Phys. Rev. A **88**(3), 032305 (2013). [CrossRef]

**30. **M. Mirhosseini, O. S. Magaňa-Loaiza, M. N. O’Sullivan, B. Rodenburg, M. Malik, M. P. J. Lavery, M. J. Padgett, D. J. Gauthier, and R. W. Boyd, “High-dimensional quantum cryptography with twisted light,” New J. Phys. **17**(3), 033033 (2015). [CrossRef]

**31. **H. Larocque, J. Gagnon-Bischoff, D. Mortimer, Y. Zhang, F. Bouchard, J. Upham, V. Grillo, R. W. Boyd, and E. Karimi, “Generalized optical angular momentum sorter and its application to high-dimensional quantum cryptography,” Opt. Express **25**(17), 19832–19843 (2017). [CrossRef]

**32. **B. Ndagano, I. Nape, B. Perez-Garcia, S. Scholes, R. I. Hernandez-Aranda, T. Konrad, M. P. J. Lavery, and A. Forbes, “A deterministic detector for vector vortex states,” Sci. Rep. **7**(1), 13882 (2017). [CrossRef]

**33. **I. Nape, E. Otte, A. Valles, C. Rosales-Guzmán, F. Cardano, C. Denz, and A. Forbes, “Self-healing high-dimensional quantum key distribution using hybrid spin-orbit Bessel states,” Opt. Express **26**(21), 26946–26960 (2018). [CrossRef]

**34. **F. Bouchard, K. Heshami, D. England, R. Fickler, R. W. Boyd, B. G. Englert, L. L. Sáchez-Soto, and E. Karimi, “Experimental investigation of high-dimensional quantum key distribution protocols with twisted photons,” Quantum **2**, 111111 (2018). [CrossRef]

**35. **F. X. Wang, W. Chen, Z. Q. Yin, S. Wang, G. C. Guo, and Z. F. Han, “Characterizing high-quality high-dimensional quantum key distribution by state mapping between different degrees of freedom,” Phys. Rev. Appl. **11**(2), 024070 (2019). [CrossRef]

**36. **G. Vallone, V. D’Ambrosio, A. Sponselli, S. Slussarenko, L. Marrucci, F. Sciarrino, and P. Villoresi, “Free-space quantum key distribution by rotation-invariant twisted photons,” Phys. Rev. Lett. **113**(6), 060503 (2014). [CrossRef]

**37. **A. Sit, F. Bouchard, R. Fickler, J. Gagnon-Bischoff, H. Larocque, K. Heshami, D. Elser, C. Peuntinger, K. Günthner, B. Heim, C. Marquardt, G. Leuchs, R. W. Boyd, and E. Karimi, “High-dimensional intracity quantum cryptography with structured photons,” Optica **4**(9), 1006–1010 (2017). [CrossRef]

**38. **J. Zhao, Y. Zhou, B. Braverman, C. Liu, K. Pang, N. K. Steinhoff, G. A. Tyler, A. E. Willner, and R. W. Boyd, “Performance of real-time adaptive optics compensation in a turbulent channel with high-dimensional spatial-mode encoding,” Opt. Express **28**(10), 15376–15391 (2020). [CrossRef]

**39. **F. Bouchard, A. Sit, F. Hufnagel, A. Abbas, Y. Zhang, K. Heshami, R. Fickler, C. Marquardt, G. Leuchs, R. W. Boyd, and E. Karimi, “Quantum cryptography with twisted photons through an outdoor underwater channel,” Opt. Express **26**(17), 22563–22573 (2018). [CrossRef]

**40. **F. Hufnagel, A. Sit, F. Grenapin, F. Bouchard, K. Heshami, D. England, Y. Zhang, B. J. Sussman, R. W. Boyd, G. Leuchs, and E. Karimi, “Characterization of an underwater channel for quantum communications in the Ottawa River,” Opt. Express **27**(19), 26346–26354 (2019). [CrossRef]

**41. **F. Hufnagel, A. Sit, F. Bouchard, Y. Zhang, D. England, K. Heshami, B. J. Sussman, and E. Karimi, “Investigation of underwater quantum channels in a 30 meter flume tank using structured photons,” New J. Phys. **22**(9), 093074 (2020). [CrossRef]

**42. **Y. Zhou, J. Zhao, B. Braverman, K. Pang, R. Zhang, A. E. Willner, Z. Shi, and R. W. Boyd, “Multiprobe time reversal for high-fidelity vortex-mode-division multiplexing over a turbulent free-space link,” Phys. Rev. Appl. **15**(3), 034011 (2021). [CrossRef]

**43. **M. P. J. Lavery, C. Peuntinger, K. Gunthner, P. Banzer, D. Elser, R. W. Boyd, M. J. Padgett, C. Marquardt, and G. Leuchs, “Free-space propagation of high-dimensional structured optical fields in an urban environment,” Sci. Adv. **3**(10), e1700552 (2017). [CrossRef]

**44. **M. P. J. Lavery, “Vortex instability in turbulent free-space propagation,” New J. Phys. **20**(4), 043023 (2018). [CrossRef]

**45. **G. Sorelli, V. N. Shatokhin, and A. Buchleitner, “Photonic orbital angular momentum in turbulence: vortex splitting and adaptive optics,” Proc. SPIE **11532**, 115320E (2020). [CrossRef]

**46. **C. Paterson, “Atmospheric turbulence and orbital angular momentum of single photons for optical communication,” Phys. Rev. Lett. **94**(15), 153901 (2005). [CrossRef]

**47. **G. A. Tyler and R. W. Boyd, “Influence of atmospheric turbulence on the propagation of quantum states of light carrying orbital angular momentum,” Opt. Lett. **34**(2), 142–144 (2009). [CrossRef]

**48. **C. Chen and H. Yang, “Characterizing the radial content of orbital-angular-momentum photonic states impaired by weak-to-strong atmospheric turbulence,” Opt. Express **24**(17), 19713–19727 (2016). [CrossRef]

**49. **C. M. Mabena and F. S. Roux, “Optical orbital angular momentum under strong scintillation,” Phys. Rev. A **99**(1), 013828 (2019). [CrossRef]

**50. **B. J. Smith and M. G. Raymer, “Two-photon wave mechanics,” Phys. Rev. A **74**(6), 062104 (2006). [CrossRef]

**51. **A. H. Ibrahim, F. S. Roux, and T. Konrad, “Parameter dependence in the atmospheric decoherence of modally entangled photon pairs,” Phys. Rev. A **90**(5), 052115 (2014). [CrossRef]

**52. **N. D. Leonhard, V. N. Shatokhin, and A. Buchleitner, “Universal entanglement decay of photonic-orbital-angular-momentum qubit states in atmospheric turbulence,” Phys. Rev. A **91**(1), 012345 (2015). [CrossRef]

**53. **Z. Wang, R. Malaney, and B. Burnett, “Satellite-to-Earth quantum key distribution via orbital angular momentum,” Phys. Rev. Appl. **14**(6), 064031 (2020). [CrossRef]

**54. **J. M. Beckers, “Adaptive optics for astronomy: principles, performance, and applications,” Annu. Rev. Astron. Astrophys. **31**(1), 13–62 (1993). [CrossRef]

**55. **R. R. Parenti, “Adaptive Optics for Astronomy,” Lincoln Lab. J. **5**, 93–113 (1992).

**56. **F. Roddier, * Adaptive optics in astronomy* (Cambridge University, 1999).

**57. **R. K. Tyson, * Principles of Adaptive Optics*, 3rd edn (CRC Press, 2011).

**58. **R. K. Tyson and B. W. Frazier, * Field guide to adaptive optics* (SPIE, 2012).

**59. **J. R. Gonzalez Alonso and T. Brun, “Recovering Quantum Information in Orbital Angular Momentum of Photons by Adaptive Optics,” arXiv preprint arXiv:1612.02552 (2016).

**60. **N. Leonhard, G. Sorelli, V. N. Shatokhin, C. Reinlein, and A. Buchleitner, “Protecting the entanglement of twisted photons by adaptive optics,” Phys. Rev. A **97**(1), 012321 (2018). [CrossRef]

**61. **G. Sorelli, N. Leonhard, V. N. Shatokhin, C. Reinlein, and A. Buchleitner, “Entanglement protection of high-dimensional states by adaptive optics,” New J. Phys. **21**(2), 023003 (2019). [CrossRef]

**62. **G. I. Taylor, “Statistical theory of turbulence,” Proc. R. Soc. A. **151**(873), 444–454 (1935). [CrossRef]

**63. **F. Asséat, R. Wilson, and E. Gendron, “Method for simulating infinitely long and non stationary phase screens with optimized memory storage,” Opt. Express **14**(3), 988–999 (2006). [CrossRef]

**64. **D. L. Fried and J. L. Vaughn, “Branch cuts in the phase function,” Appl. Opt. **31**(15), 2865–2882 (1992). [CrossRef]

**65. **Z. Tao, Y. Ren, A. Abdukirim, S. Liu, and R. Rao, “Physical meaning of the deviation scale under arbitrary turbulent strengths of optical orbital angular momentum,” arXiv preprint arXiv:2102.01891 (2021).

**66. **D. L. Fried, “Branch point problem in adaptive optics,” J. Opt. Soc. Am. A **15**(10), 2759–2768 (1998). [CrossRef]

**67. **D. L. Fried, “Adaptive optics wave function reconstruction and phase unwrapping when branch points are present,” Opt. Commun. **200**(1-6), 43–72 (2001). [CrossRef]

**68. **D. L. Fried, “Nature of the branch point problem in adaptive optics,” Proc. SPIE **3381**, 38–46 (1998). [CrossRef]

**69. **D. L. Fried, “Time-delay-induced mean-square error in adaptive optics,” J. Opt. Soc. Am. A **7**(7), 1224–1225 (1990). [CrossRef]

**70. **J. Li, Z. Zhang, J. Gao, J. Sun, and W. Chen, “Bandwidth of adaptive optics system in atmospheric coherent laser communication,” Opt. Commun. **359**, 254–260 (2016). [CrossRef]

**71. **It is worth highlighting that there is no essential difference between a conventional Hartmann wavefront sensor and a Shack-Hartmann wavefront sensor. The latter only replaces the pinhole of a conventional wavefront sensor with a microlens array (i.e., the former is pinhole-imaging, while the latter is convex-lens-imaging).

**72. **It is noted that LG modes chosen here for encoding is not the best choice, but a compromise consideration. Actually, in order to reduce the loss due to mode dependent diffraction, we commonly choose the eigenstate of *H* (i.e., the mode with the lowest loss for free-space propagation, for more details about the eigenstate and definition of *H*, we refer the reader to Refs. [73,74]) as a probe state for OAM encoding [75], which is consistent with the mathematical solution of the communication modes proposed in [76] under the circular aperture. When the Fresnel number becomes larger, the eigenstates of *H* degenerate to LG modes. Moreover, the communication implemented by LG modes can achieve a comparable performance compared to the eigenstates of *H* under weak turbulence strengths [77].

**73. **J. Zhao, M. Mirhosseini, B. Braverman, Y. Zhou, S. M. Hashemi Rafsanjani, Y. Ren, N. K. Steinhoff, G. A. Tyler, A. E. Willner, and R. W. Boyd, “Performance analysis of *d*-dimensional quantum cryptography under state-dependent diffraction,” Phys. Rev. A **100**(3), 032319 (2019). [CrossRef]

**74. **G. A. Tyler, “Spatial bandwidth considerations for optical communication through a free space propagation link,” Opt. Lett. **36**(23), 4650–4652 (2011). [CrossRef]

**75. **J. Zhao, The Institute of Optics, University of Rochester, 2020 Rochester, N. Y., 14627 (personal communication, 2020).

**76. **D. A. B. Miller, “Waves, modes, communications, and optics: a tutorial,” Adv. Opt. Photonics **11**(3), 679–825 (2019). [CrossRef]

**77. **L. Borcea, J. Garnier, and K. Sølna, “Multimode communication through the turbulent atmosphere,” J. Opt. Soc. Am. A **37**(5), 720–730 (2020). [CrossRef]

**78. **E. Karimi and E. Santamato, “Radial coherent and intelligent states of paraxial wave equation,” Opt. Lett. **37**(13), 2484–2486 (2012). [CrossRef]

**79. **E. Karimi, R. W. Boyd, P. de la Hoz, H. de Guise, J. Řeh áček, Z. Hradil, A. Aiello, G. Leuchs, and L. L. Sánchez-Soto, “Radial quantum number of Laguerre-Gauss modes,” Phys. Rev. A **89**(6), 063813 (2014). [CrossRef]

**80. **E. Karimi, D. Giovannini, E. Bolduc, N. Bent, F. M. Miatto, M. J. Padgett, and R. W. Boyd, “Exploring the quantum nature of the radial degree of freedom of a photon via Hong-Ou-Mandel interference,” Phys. Rev. A **89**(1), 013829 (2014). [CrossRef]

**81. **W. N. Plick and M. Krenn, “Physical meaning of the radial index of Laguerre-Gauss beams,” Phys. Rev. A **92**(6), 063841 (2015). [CrossRef]

**82. **J. D. Schmidt, * Numerical Simulation of Optical Wave Propagation with Examples in MATLAB* (SPIE, 2010).

**83. **M. J. Padgett, F. M. Miatto, M. P. J. Lavery, A. Zeilinger, and R. W. Boyd, “Divergence of an orbital-angular-momentum-carrying beam upon propagation,” New J. Phys. **17**(2), 023011 (2015). [CrossRef]

**84. **M. Charnotskii, “Extended Huygens–Fresnel principle and optical waves propagation in turbulence: discussion,” J. Opt. Soc. Am. A **32**(7), 1357–1365 (2015). [CrossRef]

**85. **G. Vallone, “Role of beam waist in Laguerre-Gauss expansion of vortex beam,” Opt. Lett. **42**(6), 1097–1100 (2017). [CrossRef]

**86. **Unlike other works, here, we also consider the radial-mode scrambling caused by atmospheric turbulence. Since the information is encoded by the photonic OAM state, we restrict our attention to the crosstalk probability within the OAM subspace.

**87. **G. J. Fan-Yuan, C. Wang, S. Wang, Z. Q. Yin, H. Liu, W. Chen, D. Y. He, Z. F. Han, and G. C. Guo, “Afterpulse analysis for quantum key distribution,” Phys. Rev. Appl. **10**(6), 064032 (2018). [CrossRef]

**88. **A. Ferenczi and N. Lütkenhaus, “Symmetries in quantum key distribution and the connection between optimal attacks and optimal cloning,” Phys. Rev. A **85**(5), 052310 (2012). [CrossRef]

**89. **J. M. Martin and S. M. Flatté, “Intensity images and statistics from numerical simulation of wave propagation in 3-D random media,” Appl. Opt. **27**(11), 2111–2126 (1988). [CrossRef]

**90. **G. Sedmak, “Implementation of fast-Fourier-transform-based simulations of extra-large atmospheric phase and scintillation screens,” Appl. Opt. **43**(23), 4527–4538 (2004). [CrossRef]

**91. **L. C. Andrews and R. L. Phillips, * Laser Beam Propagation through Random Media*, 2nd edn. (SPIE, 2005).

**92. **It is noted that the validity of ILPS is guaranteed by calculating its phase structure function. As discussed in Ref. [63], a great coincidence between the average structure function and theoretical one is commonly obtained by setting *N*_{col} ≥ 2.

**93. **A. Reeves, “Soapy: an adaptive optics simulation written purely in Python for rapid concept development,” Proc. SPIE **9909**, 99097F (2016). [CrossRef]

**94. **It is worth noting that the effect of atmospheric turbulence on beam propagation is wavelength dependent. We assumed the wavelength of beacon light is close enough to signal modes in this paper. Concretely, a beacon light whose wavelength around 100*nm* shorter than signal modes is acceptable [54,56,58].

**95. **G. J. Gbur, * Singular Optics*, 1st edn. (CRC Press, 2017).

**96. **D. J. Sanchez and D. W. Oesch, “Localization of angular momentum in optical waves propagating through turbulence,” Opt. Express **19**(25), 25388–25396 (2011). [CrossRef]

**97. **D. J. Sanchez and D. W. Oesch, “Orbital angular momentum in optical waves propagating through distributed turbulence,” Opt. Express **19**(24), 24596–24608 (2011). [CrossRef]

**98. **T. M. Venema and J. D. Schmidt, “Optical phase unwrapping in the presence of branch points,” Opt. Express **16**(10), 6985–6998 (2008). [CrossRef]

**99. **M. A. Herraez, D. R. Burton, M. J. Lalor, and M. A. Gdeisat, “Fast two-dimensional phase-unwrapping algorithm based on sorting by reliability following a noncontinuous path,” Appl. Opt. **41**(35), 7437–7444 (2002). [CrossRef]

**100. **J. Zhang, X. Tian, J. Shao, H. Luo, and R. Liang, “Phase unwrapping in optical metrology via denoised and convolutional segmentation networks,” Opt. Express **27**(10), 14903–14912 (2019). [CrossRef]

**101. **The realistic AO correction mentioned in this paper is the results after eliminating the wrapped cuts.

**102. **For the weak and strong scintillation regime, we set $C_{n}^{2}=2.2 \times 10^{-15} m^{-2 / 3}$ and 3.22 × 10^{−14}*m*^{−2/3}, which is equivalent to *r*_{0} = 0.05*m* and *r*_{0} = 0.01*m* when *z* = 1000*m*. It is noted that *r*_{0} is calculated with a fixed wavelength of 500*nm*, which is not same as the intrinsic wavelength of the incident states. The reason why we chose 500*nm* for the calculation is that *r*_{0} is associated with wavelength according to the Eq. (3) in Ref. [51]. In atmospheric optics, a wavelength of 500*nm* is generally specified to facilitate the use of *r*_{0} to measure the turbulence strength of a communication link.

**103. **B. Ndagano and A. Forbes, “Characterization and mitigation of information loss in a six-state quantum-key-distribution protocol with spatial modes of light through turbulence,” Phys. Rev. A **98**(6), 062330 (2018). [CrossRef]

**104. **J. A. Anguita, M. A. Neifeld, and B. V. Vasic, “Turbulence-induced channel crosstalk in an orbital angular momentum-multiplexed free-space optical link,” Appl. Opt. **47**(13), 2414–2429 (2008). [CrossRef]

**105. **It is highlighted that, in a realistic AO system, the correction order is closely associated with the number of actuator onto a deformable mirror. We set the correction capability of AO to 30-order for the realistic consideration.

**106. **S. Wang, Y. Ren, R. Rao, and X. Miao, “Influence of atmosphere attenuation on quantum interferometric radar,” Acta Phys. Sin. **66**(15), 150301 (2017). [CrossRef]

**107. **The impact of aerosol-particle scattering on the performance of OAM-based QKD can be described by adjusting the size of receiving aperture. An interesting topic is that whether eavesdroppers can steal information through scattered light (see a similar discussion in Ref. [108]).

**108. **P. Chaiwongkhot, K. B. Kuntz, Y. Zhang, A. Huang, J. P. Bourgoin, S. Sajeed, N. Lükenhaus, T. Jennewein, and V. Makarov, “Eavesdropper’s ability to attack a free-space quantum-key-distribution receiver in atmospheric turbulence,” Phys. Rev. A **99**(6), 062315 (2019). [CrossRef]

**109. **F. Wang, F. Zeng, J. Zhao, B. Braverman, Y. Zhou, M. Mirhosseini, X. Wang, H. Gao, F. Li, R. W. Boyd, and P. Zhang, “High-dimensional quantum key distribution based on mutually partially unbiased bases,” Phys. Rev. A **101**(3), 032340 (2020). [CrossRef]

**110. **Y. Yuan, D. Liu, Z. Zhou, H. Xu, J. Qu, and Y. Cai, “Optimization of the probability of orbital angular momentum for Laguerre-Gaussian beam in Kolmogorov and non-Kolmogorov turbulence,” Opt. Express **26**(17), 21861–21871 (2018). [CrossRef]

**111. **X. L. Ge, B. Y. Wang, and C. S. Guo, “Evolution of phase singularities of vortex beams propagating in atmospheric turbulence,” J. Opt. Soc. Am. A **32**(5), 837–842 (2015). [CrossRef]

**112. **G. Gbur and R. K. Tyson, “Vortex beam propagation through atmospheric turbulence and topological charge conservation,” J. Opt. Soc. Am. A **25**(1), 225–230 (2008). [CrossRef]

**113. **D. P. Greenwood, “Bandwidth specification for adaptive optics systems,” J. Opt. Soc. Am. A **67**(3), 390–393 (1977). [CrossRef]

**114. **It is noted that we generally set the bandwidth of AO as 2– 3 times the sample period of a Hartmann camera. Here we set to 5 because we consider the time delay caused by phase unwrapping.

**115. **Refs. [38,58] indicate that a comprehensive correction can be obtained if the Greenwood time of turbulence is ten times larger than the time delay of AO.