Modulation instability (MI) is a process of exponential growth of weak perturbations imposed on the continuous waves (CWs) as a result of interplay between the dispersion and the nonlinearity.^[1–5] The occurrence of MI is closely related to the soliton existence of the nonlinear system involved, a result of balancing the effects of dispersion and the nonlinearity,^[6,7] and is also responsible for the rogue wave formation.^[8–10] MI can be applied for ultrashort pulse generation and supercontinuum generation.^[11–18] In optics, MI usually occurs for the anomalous dispersion, but under some specific physical settings, MI can also occur in the normal dispersion regime.^[19–24]

In recent years, space-division multiplexing (SDM) systems based on the few mode and multicore fibers have been proposed as a promising technology to increase the transmission capacity of optical fibers, and has attracted worldwide interests.^[25–28] Nonlinear effects in these fibers can significantly affect the performance of the SDM systems, and thus have received much attention in recent years.^[29–32] There are, however, very few studies investigating the MIs in such fibers except for some MI analysis in high-birefringence multimode fibers^[33] and two- and three-core fibers.^[34–36]

In this paper, we give a detailed MI analysis of RB-TMFs by taking the effects of DMGD, GVD ratio, and power ratio between the two modes, which are significant parameters in multimode fibers and has never been thoroughly studied, into consideration in both normal and anomalous dispersion regimes. The main findings are listed below.

The MI analysis in this paper is finally verified by the wave propagation method.

The wave propagation in the randomly birefringent nonlinear multi-mode optical fibers in the weakly coupling regime, where the modes coupling due to various external factors are weak such as the coupling between LP₀₁ and LP_11a modes, are derived in detail in Ref. [31]. In this paper, we study the wave propagation in such a TMF, and excite the LP₀₁ and LP_11a modes only. The corresponding governing equations derived in Ref. [31] degenerate into

We normalize Eqs. (1)–(4) into the following equations

The continuous wave (CW) solutions of Eqs. (5)–(8) follow

If the solution K of the dispersion relation [Eqs. (24)–(26)] with respect to Ω is complex, the CWs will grow exponentially and the MI gain is defined as

With s = r = ±1, δ₁ = δ₀ = 0, f_AB = 0, and f_BB = 1, the modes are uncoupled, and the equation models of Eqs. (5)–(8) degenerate into two identical sets of Manakov equations with either set representing an RB-SMF. For either mode, the dispersion relation of Eqs. (24)–(26) degenerate into

For the general case, the dispersion relation of Eqs. (24)–(27) is quite involved, and can be only treated numerically. The MI analysis for the general case is studied in detail in the following sections.

As a benchmark, we first study the MI in the RB-SMFs. From the dispersion relation of Eq. (29), we see that there is no MI in the normal dispersion regime (r = 1), and MI only exists in the anomalous dispersion regime (r = −1) with the gain

As expected, the instability gain increases with P as shown in Fig. 1. Note that the gain spectra for the negative and positive perturbation frequencies are totally symmetric, and thus only the MI spectrum for the positive perturbation frequency is shown in Fig. 1.

Fig. 1.

	Figure Option ViewDownloadNew Window
	Fig. 1. Variation of MI spectra with total power in an RB-SMF.

We next show MI results in the RB-TMFs in both normal and anomalous dispersion regimes. As in the RB-SMFs (Fig. 1), our results show that the gain spectra in the RB-TMFs for negative and positive perturbation frequencies are also symmetric. Therefore, we only show the MI spectra with the positive perturbation frequency part in the following section.

4.1. Normal dispersion regime

We first consider the normal dispersion. The DMGD plays a crucial role. In the absence of DMGD, as in the RB-SMFs, no MI exists in the RB-TMFs. In the presence of DMGD, MI can be generated in the RB-TMFs.

Figure 2 shows the variation of MI gain with total power P for r = 1, s = 1.2, f_AB = 0.53, f_BB = 0.72, δ₁ = 2, P₁/P₂ = 1, and P = P₁ + P₂. As the total power increases, this MI band enhances, reduces and finally vanishes after a critical total power. This result is understandable. Physically, DMGD tends to split the pulse, while the nonlinear effects are to suppress the pulse splitting. When the nonlinear effects are strong enough, the effect of DMGD should be completely suppressed, and thus the MI should vanish as the total power exceeds a critical total power.

Fig. 2.

	Figure Option ViewDownloadNew Window
	Fig. 2. Variation of MI spectra with total power in an RB-TMF for r = 1, s = 1.2, fAB = 0.53, fBB = 0.72, δ1 = 2, P1/P2 = 1, and P = P1 + P2.

Figure 3 shows the variation of MI gain with DMGD for r = 1, s = 1.2, f_AB = 0.53, f_BB = 0.72, P = 1.5, and P₁/P₂ = 1. For a given total power, no MI is present for small δ₁. As δ₁ increases, MI appears, enhances distinctively and eventually saturates with the modulation frequency shifting to the higher frequency at the same time. We remark that the role of DMGD in RB-TMFs is similar with the polarization mode dispersion (PMD) in highly birefringent (Hi-Bi) SMFs,^[39,40] which is expected as both the DMGD and PMD denote the group delay only that DMGD denotes the group delay between two distinct spatial modes but PMD denotes the group delay between two polarization components of a given mode.

Fig. 3.

	Figure Option ViewDownloadNew Window
	Fig. 3. Variation of MI spectra with DMGD in an RB-TMF for r = 1, s = 1.2, fAB = 0.53, fBB = 0.72, P = 1.5, and P1/P2 = 1.

Figure 4 shows the variation of MI gain with power ratio (P₁/P) for r = 1, s = 1.2, f_AB = 0.53, f_BB = 0.72, P = 1.5, and δ₁ = 2. When the power is totally launched in the LP_11a or the LP₀₁ mode, namely P₁/P = 0 or 1, no MI exists for any values of DMGD. As power ratio increases from 0, the MI appears and the MI band broadens with its gain increasing and approaching to maximum at P₁/P = 0.5, i.e., equal power distribution between the two modes. Across P₁/P = 0.5, the MI of this band changes in the opposite way. We emphasize that only the power distribution between the two modes can affect the MI spectra, and the power distribution in the x- and y-polarizations of each mode does not affect the MI spectra in the RB-TMFs.

Fig. 4.

	Figure Option ViewDownloadNew Window
	Fig. 4. Variation of MI spectra with the power ratio between the LP01 and LP11a modes in an RB-TMF for r = 1, s = 1.2, fAB = 0.53, fBB = 0.72, δ1 = 2, and P = 1.5.

Figure 5 shows the variation of MI gain with GVD ratio (s = β_2B/|β_2A|) for r = 1, f_AB = 0.53, f_BB = 0.72, δ₁ = 2, P = 1.5, and P₁/P₂ = 1. Generally, GVD ratio between the LP_11a and LP₀₁ modes reduces the instability gain of the MI band, and shrinks the modulation frequency range at the same time. Such effects are dramatic for relatively small values of GVD ratio as shown in Fig. 5.

Fig. 5.

	Figure Option ViewDownloadNew Window
	Fig. 5. Variation of MI spectra with the GVD ratio between the LP11a and LP01 modes in a RB-TMF for r = 1, fAB = 0.53, fBB = 0.72, δ1 = 2, P = 1.5, and P1/P2 = 1.

The dependence of the critical total power (P_cr) for the vanishing of the MI in Fig. 2 on the DMGD (δ₁), power ratio (P₁/P) and GVD ratio (s) is shown in Fig. 6. This critical total power increases significantly as DMGD increases (Fig. 6(a)), and decreases in a dramatic manner with the increment in the GVD ratio (Fig. 6(c)). The dependence of P_cr on the power ratio P₁/P is relatively weak except at P₁/P = 0 and P₁/P = 1, where the critical power vanishes, namely P_cr = 0.

Fig. 6.

	Figure Option ViewDownloadNew Window
	Fig. 6. The dependence of the critical total power on (a) DMGD for r = 1, s = 1.2, fAB = 0.53, fBB = 0.72, and P1/P2 = 1; (b) power ratio for r = 1, s = 1.2, fAB = 0.53, fBB = 0.72, and δ1 = 2; (c) GVD ratio for r = 1, fAB = 0.53, fBB = 0.72, δ1 = 2, and P1/P2 = 1.

4.2. Anomalous dispersion regime

We next consider the anomalous dispersion. It is practical for the value of DMGD being zero by special fiber design.^[37,38] Figure 7 shows the variation of MI with the total power in an RB-TMF in the absence of DMGD for r = −1, s = −1.2, f_AB = 0.53, f_BB = 0.72, δ₁ = 0, and P₁/P₂ = 2/3. Without the effect of DMGD, there is only one MI band. Compared to the corresponding RB-SMFs (Fig. 1), MI gain in the RB-TMFs reduces and the MI band also shrinks. Quantitatively, for the given parameters in Fig. 7, the gain and the instability bandwidth are decreased by 20% and 15% respectively when a same total power that is launched into an RB-SMF is now launched into an RB-TMF.

Fig. 7.

	Figure Option ViewDownloadNew Window
	Fig. 7. Variation of MI spectra with the power ratio between the LP01 and LP11a modes in an RB-TMF for r = −1, s = −1.2, fAB = 0.53, fBB = 0.72, δ1 = 0, and P1/P2 = 2/3.

In the presence of DMGD, MI spectra is distinctly different as shown in Fig. 8, where the parameters are the same as those in Fig. 7 but δ₁ = 2. There exists a critical total power. As power increases from zero, two MI bands appear, enhance, and finally merge into one band across this critical total power. This is also understandable in terms of the physics that DMGD tends to split the pulse and the nonlinear effect tends to suppress the pulse splitting. Therefore, the effect of DMGD can be suppressed as the power increases and the two MI bands should merge into one when power is large enough for a given DMGD. On the other hand, in comparison with the MI in RB-SMFs (Fig. 1), the MI gain in RB-TMFs is decreased further in the presence of DMGD. For δ₁ = 2, the gain in RB-TMFs shown in Fig. 8 is decreased by 40% compared with that in Fig. 1.

Fig. 8.

	Figure Option ViewDownloadNew Window
	Fig. 8. Variation of MI spectra with the power ratio between the LP01 and LP11a modes in an RB-TMF for r = −1, s = −1.2, fAB = 0.53, fBB = 0.72, δ1 = 2, and P1/P2 = 2/3.

Figure 9 shows the variation of MI with DMGD for r = −1, s = −1.2, f_AB = 0.53, f_BB = 0.72, P = 1, and P₁/P₂ = 2/3. Generally, DMGD reduces the MI gain. On the other hand, a new MI band in the low frequency is generated after a critical DMGD, and the original MI band rapidly shifts to the high frequency as DMGD increases. The effect of DMGD is also similar with that of PMD in the anomalous dispersion regime. The same arguments on effect of DMGD in the normal dispersion regime can be also applied here.

Fig. 9.

	Figure Option ViewDownloadNew Window
	Fig. 9. Variation of MI spectra with DMGD in an RB-TMF for r = −1, s = −1.2, fAB = 0.53, fBB = 0.72, P = 1, and P1 = P2 = 2/3.

Figure 10 shows the variation of MI with the power ratio (P₁/P) between the two modes for r = −1, s = −1.2, f_AB = 0.53, f_BB = 0.72, δ₁ = 2, and P = 1. At P₁/P = 0 (or 1), i.e., the total power is totally launched in LP_11a (or LP₀₁) mode, there is only one MI band. As power ratio (P₁/P) deviates from 0 (or 1), a new high frequency MI band appears, and the gains of both MI bands changes dramatically with P₁/P. Specifically, as the power ratio increases from 0, the MI gain of the low frequency band reduces, reaches its minimum value at around P₁/P = 0.4, and then rapidly enhances. In contrast, the high frequency band enhances first, begins to reduce after about P₁/P = 0.4 and finally vanishes at P₁/P = 1. From Fig. 10, we also find that the dominant MI band can be switched between the two MI bands. Quantitatively, the high (low) frequency band dominates the MI for P₁/P = 0.4 ∼ 0.6 (P₁/P < 0.4 and P₁/P > 0.6).

Fig. 10.

	Figure Option ViewDownloadNew Window
	Fig. 10. Variation of MI spectra with the power ratio between the LP01 and LP11a modes in an RB-TMF for r = −1, s = −1.2, fAB = 0.53, fBB = 0.72, δ1 = 2, and P = 1.

Figure 11 shows the variation of MI with GVD ratio for r = −1, f_AB = 0.53, f_BB = 0.72, δ₁ = 2, P = 1, and P₁/P₂ = 2/3. The effect of the GVD ratio is to shift the two MI bands to the low frequency range as the magnitude of GVD ratio increases. At the same time, GVD ratio enhances the gain of the high frequency band as the increment in the magnitude of GVD ratio, but imposes slight effect on the gain of the low frequency band.

Fig. 11.

	Figure Option ViewDownloadNew Window
	Fig. 11. Variation of MI spectra with the GVD ratio between the LP11a and LP01 modes in an RB-TMF for r = −1, fAB = 0.53, fBB = 0.72, δ1 = 2, P = 1, and P1/P2 = 2/3.

Figure 12 shows the dependence of the critical total power in Fig. 8 on DMGD, power ratio and GVD ratio respectively. The critical total power increases monotonically as DMGD increases, and reduces monotonically as the increment of GVD ratio. However, the critical total power varies in a complicate way with the power ratio. As for the normal dispersion, the critical total power is also zero, i.e., P_cr = 0, at P₁/P = 0 or P₁/P = 1. As P₁/P increases from 0, P_cr is nonzero and experiences a series of reduction and increment. In general, the dependence of the critical power on the DMGD is most dramatic (compare Fig. 12(a) with Fig. 12(b) and Fig. 12(c)).

Fig. 12.

	Figure Option ViewDownloadNew Window
	Fig. 12. The dependence of the critical total power on (a) DMGD for r = −1, s = −1.2, fAB = 0.53, fBB = 0.72, and P1/P2 = 1; (b) power ratio for r = −1, s = −1.2, δ1 = 2, fAB = 0.53, and fBB = 0.72; (c) GVD ratio for r = −1, δ1 = 2, fAB = 0.53, fBB = 0.72, and P1/P2 = 1.

According to above analysis to the effect of DMGD (Figs. 7 and 9), power ratio (Fig. 10), and GVD ratio (Fig. 11), the MI in RB-TMFs, in comparison with RB-SMFs (Fig. 1), has the following distinctive characteristics: first, the MI gain is decreased, the decrement depends on the specific fiber parameters, and the minimum gain can be only 1/4 of that in an RB-SMF for a given total power; second, a new MI band in the high frequency can be generated in the presence of DMGD.

To verify the MI analysis above, we solve the four coupled nonlinear Schrödinger equations of Eqs. (5)–(8) numerically by launching CWs directly into each polarization of the LP₀₁ and LP_11a modes, i.e.,

We first consider the case of the normal dispersion. The parameters used in the simulation are taken as follows: r = 1, s = 1.2, f_AB = 0.53, f_BB = 0.72, δ₁ = 2, P = P₁ + P₂ = 1.5, P₁ = P₂ = 0.75, P_1x/P₁ = 0.4, P_1y/P₁ = 0.6, P_2x/P₂ = 0.7, and P_2y/P₂ = 0.3. Figure 13 shows the CW propagation in an RB-TMF with above parameters. At about ξ = 30, a train of periodic waves appear with the period of 4.59, which corresponds to the frequency of 0.22. The calculated optimum modulation frequency generating the maximum gain for above parameters selected as shown by the green dash line in Fig. 4(b) is 0.22, in quite good agreement with the results of numerical simulation. As further propagation, the periodic waves vanish and become very irregular, which justify the existence of multi modulation frequency components as predicted in the MI analysis.

Fig. 13.

	Figure Option ViewDownloadNew Window
	Fig. 13. (a) The evolution of the MI from a CW input calculated for the normal dispersion with r = 1, s = 1.2, fAB = 0.53, fBB = 0.72, δ1 = 2, P = P1 + P2 = 1.5, P1 = 0.75, P2 = 0.75, P1x/P1 = 0.4, P1y/P1 = 0.6, P2x/P2 = 0.7, and P2y/P2 = 0.3; (b) the waves in the x- and y-polarizations of each mode at ξ = 0 (red line) and ξ = 30 (blue line).

We then consider case of the anomalous dispersion. Figure 14 shows the CW propagation in an RB-TMF for taking the following parameters: r = −1, s = −1.2, f_AB = 0.53, f_BB = 0.72, δ₁ = 2, P = P₁ + P₂ = 1, P₁ = 0.9, P₂ = 0.1, P_1x/P₁ = 0.4, P_1y/P₁ = 0.6, P_2x/P₂ = 0.1, and P_2y/P₂ = 0.9. At about ξ = 13, a train of the periodic waves appears with the period of 5.08, corresponding to the frequency of 0.20, which agrees quite well with the calculated optimum modulation frequency of 0.20 for the parameters selected as shown with the blue dash line in Fig. 10(b). As the wave propagation distance increases, the periodic waves become very spiky, which also verify the existence of the multi-modulation frequency components.

Fig. 14.

	Figure Option ViewDownloadNew Window
	Fig. 14. (a) The evolution of MI from a CW input calculated for the anomalous dispersion with r = −1, s = −1.2, fAB = 0.53, fBB = 0.72, δ1 = 2, P1 = 0.9, P2 = 0.1, P1x/P1 = 0.4, P1y/P1 = 0.6, P2x/P2 = 0.1, and P2y/P2 = 0.9; (b) the waves in the x- and y-polarizations of each mode at ξ = 0 (red line) and ξ = 13 (blue line).

We remark that the selection of random number as the noise is valid.^[41,42] On the one hand, the bandwidth of the noise generated by the random numbers has been wide enough to cover all the possible modulation frequencies for the parameters selected in our numerical simulations. The frequencies of the noise in the simulation have covered a range of about ±30 around the optical carrier frequency, while the modulation frequencies calculated in the MI analysis are in a range of about ±0.5 around the optical carrier frequency. On the other hand, the results of our numerical simulation fit well with the MI analysis as discussed above.

In this paper, we studied the MIs in RB-TMFs systematically by solving four coupled nonlinear Schrödinger equations with DMGD and GVD ratio between the two modes. The effects of power ratio between the two modes are also studied. New MI characteristics are observed in both normal and anomalous dispersion regimes.

In the normal dispersion regime, MI, which is absent in RB-SMFs, can occur in the RB-TMFs for any total power smaller than a critical value under the effect of DMGD. The maximum gain is obtained for equal power distribution between the two modes, while the power distribution in the two polarization directions in each mode does not affect the MI. The effect of GVD ratio is to reduce the MI gain and shrink the MI band.

In the anomalous dispersion regime, in comparison with the RB-SMFs, the MI is generally reduced. On the other hand, there also exists a critical total power in the presence of DMGD. Two MI bands are present for the total power smaller than this critical total power, and the dominant MI band can be switched between the two bands by adjusting the power distribution in the two modes. GVD shifts the modulation frequencies of the two bands to the lower frequency. On the other hand, the GVD ratio enhances the gain of the high frequency band, but has little effect on the MI gain of the low frequency band.

MIs in the RB-TMFs are much richer than those in the RB-SMFs. Our results should be quite useful for the evaluation of the nonlinear effects in SDM systems and other applications based on multimode fibers. The effects of the higher order dispersion and higher order nonlinearity in RB-TMFs will be explored in our future studies.