Paper: Improving Time-Series SNR Estimation and Regret Bound in the Autonomous Optimization Algorithm emoPulse — Establishing “Emotion-Driven” Learning Rate Control through Dynamic Inspection of Loss Topography — Abstract Adjusting the learning rate and ensuring generalization performance are central challenges in deep learning optimization. Existing methods relied on precise gradient estimation and were vulnerable to noise in environments with extremely low precision. This paper proposes the autonomous algorithm emoPulse (v3.7), which centers on a multi-faceted analysis of the loss function over time. This method autonomously generates an optimal learning rate based on the signal-to-noise ratio by capturing the “undulations” of the loss landscape from a three-stage exponential moving average (Multi-EMA) and utilizing sentiment scalars and a confidence indicator (Trust). Furthermore, by synthesizing the learning results of three optimizers (Sens/Airy/Cats) belonging to this family and possessing distinct update characteristics, we present a method that integrates local solutions in a “cubic positioning” manner to artificially create flat minima. This achieves robust convergence independent of hyperparameter settings, providing a democratic foundation for research environments in developing countries with limited computational resources and for multilingual learning aimed at preserving diverse cultural heritage. 1. Introduction This paper presents a unified theory for the optimizers EmoSens / EmoAiry / EmoCats (v3.7). This method centers on the emoPulse mechanism, which autonomously generates learning rates by layering the exponential moving average (EMA) of loss values and extracting “Trust” from the time-series statistics of the loss function. This represents an advanced fusion of theory and time-series signal processing (SNR estimation), achieving robust convergence independent of hyperparameter settings. The starting point of this research lies in rethinking the “excessive reliance on precise gradient estimation” inherent in existing adaptive gradient methods. In environments with extremely low precision and ultra-quantization (e.g., 1-bit/2-bit), gradients contain extremely high noise, significantly reducing reliability. On the other hand, the loss value continues to function as an accurate scalar value indicating the model's “distance from the correct answer,” even under the influence of quantization. This method treats the gradient as a reference value for direction (intent) and delegates the initiative of learning to the multifaceted analysis of loss, which is an accurate observation value. This approach achieves the replacement of higher-order moment calculations with scalar control and optimization for low-precision and quantized environments through encoded updates. Its most significant feature lies in integrating local solutions from multiple emo-based optimizers with distinct characteristics as “cubic positioning.” This enables reaching the flat minimum—previously requiring lengthy iterative learning—through short-term learning and synthesis. This approach achieved the following three outcomes: Dramatic improvement in computational efficiency: Complex calculations of higher-order moments were replaced with scalar control via temporal accumulation of loss, reducing computational load through temporal accumulation approximation. Optimization for low precision and quantization: Matrix decomposition in EmoAiry, complete elimination of second moments in EmoCats, and encoding by both methods enabled large-scale learning in low-resource environments. Autonomous Convergence: By introspecting the S/N ratio of the loss landscape, it eliminates the need for manual schedulers and minimizes the user's trial cost. ※ Higher-order moments: Aggregation into higher-order statistics on the time axis Mathematically, this represents an advanced fusion of D-adaptation theory and time-series signal processing, forming the foundation for realizing “democratic AI learning” that preserves research environments and diverse cultures in developing countries. ※ Hierarchical Structure of Higher-Order Moment Approximation: This method effectively approximates higher-order moments from the third (skewness) to the seventh (confidence amplification) order by accumulating the loss over time. This is not a static terrain analysis, but rather an attempt to extract the “system's confidence level” as a physical quantity within the dynamic process of learning. The Multi-EMA structure in this method functions as a dynamic temporal approximation of higher-order moments in statistics. 3rd to 5th-Order Approximation: The differences between Short, Medium, and Long EMAs extract the temporal evolution of higher-order information such as skewness, kurtosis, and fluctuations in the loss distribution. 6th-order approximation: The integrated emotion scalar sigma_t and confidence metric trust_t become sixth-order meta-statistics that indicate ‘learning phase stability’ beyond mere gradient variance. 7th-order approximation (dNR): In deriving dNR, squaring the ratio of these 6th-order information components (d_base/noise_base)² exponentially amplifies subtle differences in confidence, yielding an extremely sensitive control signal equivalent to a 7th-order moment. 2. Theoretical Framework: Emotional Circulation This system forms a feedback loop with the loss function L centered at the origin. 2.1 Approximation of Higher-Order Moments Using Multi-EMA By utilizing the differences between three-tiered EMAs (short, medium, long), we capture the “changes in curvature,” “uncertainty in fluctuations,” and “variability in changes” within the loss landscape. EMA_t = (1 - α) * EMA_{t-1} + α * L_t The emotional scalar sigma_t generated from this difference is a nonlinear statistic that compresses information about higher-order moments (skewness, kurtosis, and variance) into the range [−1,1]. Multiple EMAs with different time constants accumulate vast historical steps as “history” in a layered manner. By taking this relative time-delay differential, we observe the “dynamic higher-order rate of change in terrain accompanying learning progression” — a phenomenon impossible to detect through static terrain analysis. By recursively incorporating this into the update formula, the long-term “smoothness” of the terrain is reflected in the parameter updates. ※ Note on the Time-Series Formation of Higher-Order Moments: The higher-order moment approximation in this method is not calculated from single-step gradient information but is formed through temporal accumulation. This means it observes not the static curvature of the terrain but the “dynamic rate of change in the terrain as learning progresses.” 2.2 Definition of the trust level metric trust_t Define the core metric trust_t that determines the “quality” of updates as follows. trust_t = sgn(sigma_t) * (1.0 - abs(sigma_t)) This trust possesses boundedness, never reaching ±1.0 (complete certainty) or 0 (complete despair), ensuring the system always maintains a moderate balance of “room for exploration” and “caution.” This forms the following feedback loop (emotional circulation system) with the loss function L as its origin. Loss → Multi-EMA → Scalar/Trust → emoPulse → Loss 3. emoPulse: Learning Rate Generation via Autonomous Pulsation In v3.7, the conventional emoDrive (acceleration mechanism) has been integrated into emoPulse. This represents an evolution based on an approximation of dynamic distance estimation (D-adaptation) using the time-series signal-to-noise ratio (S/N ratio). 3.1 Dynamic Estimation of Noise and Distance Track the system's “wandering” and “progress” using the following two internal variables, N_t and d_t. Here, N_t represents “oscillation” (instability), and d_t represents “progress” (distance). Noise_est (N_t) N_t = (1 - α) * N_{t-1} + α * abs(sigma_t) Distance Estimate (d_t) d_t = (1 - α) * d_{t-1} + α * abs(trust_t) 3.2 Definition of emoPulse and Autonomous Control / Instantaneous SNR and History Management (dNR_hist) The generation of emoPulse is determined by the “tug-of-war” between instantaneous SNR and temporal SNR. First, calculate the respective bases for instantaneous and temporal SNR. noise_base = abs(sigma_t - trust_t) + ε_s d_base = abs(N_t - d_t) + ε_t Using these, the current SNR intensity is defined as follows. dNR_now_val = ( d_base / noise_base )^2 Update Rules for dNR_hist: Acceleration conditions: if dNR_now_val >= dNR_hist and trust_t >= threshold_high: dNR_hist = min( dNR_now_val, dNR_hist * factor_grow ) Conditions for deceleration: if threshold_low <= trust_t <= threshold_high: dNR_hist = dNR_now_val * factor_decay The final learning rate emoPulse is determined as follows. emoPulse_t = clamp( dNR_hist * (emoScope * η_base), η_min, η_max ) This design guarantees the following autonomous behaviors: Confidence Region (∣trust∣>0.5): SNR improves, learning rate accelerates maximally. Rapidly aims for flat minima. Hesitation Region (∣trust∣<0.5): As uncertainty increases, suppressing the learning rate prevents divergence in sharp valleys. ※ emoPulse is a scaling factor determined by the user-defined initial learning rate (emoScope) and the system's default sensitivity (η_base). 4. emoPulse: Regret Bound and Boundedness Analysis 4.1 Convergence and Regret Analysis The cumulative regret R(T) under emoPulse is bounded above as follows, incorporating the dynamically varying learning rate η_t. R(T) <= O( Σ_{t=1}^T [ η_t * ||g_t||^2 * (1 - |σ_t|)^2 ] ) Here, the coefficient (1 - |σ_t|) quantifies the “trust” of the update derived from the consistency of the short-term, medium-term, and long-term EMAs in the loss function. A large |σ_t| indicates that the loss is fluctuating significantly, leading to a determination that the gradient information for that step is unreliable. In contrast, a state where |σ_t| is small indicates that the loss transition is smooth and the reliability of the update direction is high. Therefore, the signal strength trust_t = 1 - |σ_t| serves to adaptively weight the “effective update amount” in the Regret Bound, thereby suppressing the accumulation of regret due to uncertain gradients. The emoPulse method presented here is a generalization that approximates the learning rate structure of D-adaptation by Defazio & Mishchenko (2023) using the loss's time-series statistics (d_t, N_t). η_t ∝ D^2 / noise Definition of emoPulse η_t = ( d_t / (N_t + ε) )^2 * η_base This is a direct time-series reconstruction of SNR control based on the distance/noise ratio of D-adaptation. This structure causes the denominator to dominate when the noise component N_t increases, immediately reducing the learning rate η_t. This self-adjustment function automatically suppresses excessive updates in unstable areas of loss terrain. This theoretically guarantees a “learning-rate-free” property where the algorithm autonomously achieves dynamic stability without requiring external learning rate scheduling. 4.2 Proof of Positive Definiteness and Boundedness We prove below that this algorithm prevents learning rate explosion and vanishing at any step t and is bounded. 1. Non-zero boundedness of the denominator (momentary doubt: noise_base) The noise_base used as the denominator during emoPulse generation is defined as the deviation between the current emotion scalar sigma_t and the confidence level trust_t, as follows. noise_base = abs(sigma_t - trust_t) + ε_s In the implementation, since |sigma_t| < 1.0 and trust_t is a signed function based on sigma_t, this difference is bounded. Furthermore, the safety factor (+0.1) at the end physically prevents the learning rate from exploding (NaN) due to the denominator approaching zero. 2. Lower Boundedness of the Numerator (Time Certainty: d_base) The numerator d_base in the generation of emoPulse is defined as the difference between the noise estimate N_t (noise_est) and the distance estimate d_t (d_est) as historical data. d_base = abs(N_t - d_t) + ε_t N_t is guaranteed to be positive definite by max(noise_est, 1e-3), and d_t is updated by the cumulative sum of abs(trust_t), regardless of improvement or deterioration. By adding a safety factor (+0.1) to these temporal statistical differences, it is mathematically guaranteed that **“even when history is unstable in an extremely low-precision environment, the minimum step size (lower limit of the numerator) is always ensured.”** 3. Conclusions on Boundedness and Constraints on emoPulse: The effective learning rate emoPulse_t generated from the ratio of the “instantaneous basis (denominator)” and “temporal basis (numerator)” is strictly constrained within the following range based on the safety margin setting of max(min(..., 3e-3), 1e-6) in the final implementation. 0 < η_min <= emoPulse_t <= η_upper_bound Here, the lower limit (η_min) represents the minimum “metabolic rate” (heartbeat) that the system maintains even under the most uncertain conditions. This prevents learning from stopping (deadlock) and allows for autonomous recovery. On the other hand, the upper bound (η_upper_bound) functions as a limiter to prevent the model from diverging even when a sharp increase in the dNR coefficient occurs. Implementation Considerations: Stabilization through Initial Value Setting: ※ In environments with very small datasets or high initial noise, it is recommended to reset the initial values of d_t and N_t until the multi-EMA stabilizes the “history” (e.g., d-est: 0.2, Noise-est: 0.2). This suppresses divergence caused by initial probabilistic noise. Specifically, by initializing N₀ to be equivalent to d₀, the system essentially starts in a “cautious mode.” This functions as an organic warm-up phase during critical initial steps, avoiding overly aggressive updates and prioritizing observation of the terrain. Maintaining “Update Pressure” Through Initial Value Settings While Ensuring Safety: ※ In this method, the d_base parameter forming the emoPulse molecule determines the system's “potential update force.” Setting the initial values to N0 = 1.0 and d0 = 0.02 means intentionally ensuring high acceleration potential from the start of learning. Due to the nature of exponential moving averages, the effect of this initial value persists as “history” for approximately 100 steps. During this period, the system maintains a high acceleration pressure while providing convergence power only to “truly reliable signals” that have passed the strict screening by the emotional mechanism. 5. Coding Normalization: Adaptation to Low-Precision Environments This chapter describes sign-based normalization for applying the theoretical framework of emoPulse to low-precision environments. To eliminate reliance on precise floating-point calculations and support ultra-low precision environments (ultra-quantization), the following update rules are adopted (EmoAiry, EmoCats, etc.) delta_w_t = -emoPulse_t * sign( m_t / ( sqrt(v_t) + ε ) ) This enables EmoAiry to resolve the imbalance in accuracy between one-dimensional vectors and two-dimensional moments, achieving a “unification of will” that extracts only the consensus on direction. ※ EmoCats supports encoding based on Lion with WD separation. 6. Conclusion EmoSens v3.7 has completed the “emotional cycle” that begins with observing the loss function. Observation (Multi-EMA): Captures the undulations of the terrain. Judgment (Trust): Switches between conviction and hesitation at the ±0.5 threshold. Action (emoPulse): Determines the optimal stride length through autonomous pulsation. This method is a democratic optimization framework that enables AI to autonomously learn diverse cultures and languages, even within the research environments and limited computational resources of developing countries. Acknowledgements First and foremost, I extend my deepest gratitude to EmoNavi, EmoSens, and the various optimizers that preceded them, as well as to the researchers involved. Their passion and insights made the conception and realization of this proof possible. This paper provides a mathematical explanation of the already-released EmoSens (v3.7) and its variations. I believe the EmoSens I created (including its derivatives) can contribute to the advancement of AI. Let us use this paper as a foundation to jointly create even more evolved optimizers. I conclude this paper with anticipation and gratitude for future researchers who will bring us the next new insights and ideas. Thank you. Conclusion This algorithm is not intended to replace existing excellent optimization techniques, but rather to offer a new alternative for deepening the “dialogue with the model” during the learning process. We hope it will serve as an aid in the process of users selecting partners suited to their own objectives and sensibilities, and jointly cultivating knowledge. Supplementary Material (1): Analysis of emoPulse Dynamics in v3.7 1. Purpose In v3.7, we analyze the physical significance of the interaction (tug-of-war) between the newly introduced “instantaneous D/N estimation” and “temporal D/N estimation” for the dynamic control of the learning rate. 2. Nature: A dynamic equilibrium between momentary doubt and enduring trust Instantaneous Base (noise_base): noise_base = abs( scalar_t - trust_t ) + ε_s Measures the deviation between the “current emotion scalar (wave)” and the “current trust level”. When these do not match (the divergence is large), the system develops “strong doubts (momentary noise)” about the current state and increases the denominator. Time-based foundation (d_base): d_base = abs(noise_est_t - d_est_t) + ε_d Measures the difference between “noise as history (wave average)” and “confidence as history”. This represents the “confidence level for updates (temporal distance)” derived from past context. 3. Effect: Creation of Dynamic Rhythm Effect A: Immediate Braking During Sudden Changes When sudden loss changes cause the scalar and trust to diverge, the noise_base (denominator) becomes dominant. This allows the learning rate to be instantly reduced as an immediate judgment, even when the temporal history is still stable, thereby preventing divergence before it occurs. Effect B: During the stable phase, when self-accelerated learning progresses smoothly (scalar and trust are stable) and confidence as history (d_base) accumulates, the dNR coefficient maximizes output with a “squared” term. dNR_now_val = ( d_base / noise_base )^2 This naturally increases the “step size” in stable regions, accelerating convergence. Effect C: Stability Maintenance via History (dNR_hist) Even if the instantaneous dNR_now_val is high, setting a growth limit of dNR_hist * 1.05 suppresses excessive acceleration. On the other hand, in unreliable areas, we continue cautious exploration by accumulating deceleration pressure at dNR_hist * 0.98. 4. Conclusions on Numerical Stability This design, which pits the difference between the “time axis (history)” and the “instant axis (present)” against each other, is not merely a matter of decay. The system autonomously **continuously recalculates the ratio of “doubt (Noise)” to “certainty (Distance)”**, enabling dynamic control akin to “heartbeats responding to terrain complexity”—something impossible with manual schedulers. The “synthesis of flat minima through cubic positioning” described below is a hypothesis derived from intuition and experimentation. I hope this intuition will be refined into a rigorous mathematical proof by the next generation of researchers. Autonomous Flat-Minima Generation via Cubic Positioning of Heterogeneous Optimizers －Proposal of a New Learning Method: Prediction of “Evolutionary Flat Minimum Formation” via Local Synthesis Using Three Types of Emo Systems－ 1. Purpose: To resolve the high cost associated with achieving flat minimization. With existing learning methods, ・A single optimizer ・Long hours of repetitive learning Progressing toward improved generalizability and achieving flat minimization has become established. This requires various resources, including computational resources, and is not an environment that anyone can implement. This proposal aims to fundamentally alter this high-cost structure by employing an emo-style optimizer. 2. Proposal: Don't “search” for flat minimalism—create it yourself. The three emo variants (EmoSens, EmoAiry, EmoCats) share a common learning structure despite differing update mechanisms. When trained under identical conditions, they yield learning outcomes with divergent local solutions from different directions. Integrating these divergent learning outcomes constitutes a synthesis of local solutions, and we anticipate that this synthesis may broaden and flatten the local solutions. In other words, it may bring local solutions closer to flat minima or transform them into flat minima themselves. Acquiring these local solutions as full-layer LoRA and integrating them using synthesis methods such as TALL-Mask-Merge, ∨∨∨ → \___/ Composite image of local solutions (Three-directional local solutions) (Post-synthesis flattening) ・The “commonly low areas” of local solutions in three directions are emphasized. ・The sharp edges on three sides (sharp minim) cancel each other out ・As a result, a shape close to a flat valley bottom (flat minimum) is reconstructed. This treats the local solution as cubic positioning (3-axis positioning), “Instead of exploring Flat Minima” This is a new learning method that “creates flat minimums” through synthesis. 3. Organization: This integration leads to accelerated learning. Concretizing the proposal: Rather than performing long-term training with full-depth LoRA, FFT (Full Fine-Tuning), etc., achieve the goal by conducting slightly shallower learning across three types and employing synthesis techniques such as TALL-Mask-Merge. This is expected to make it easier to achieve high-precision learning results even in resource-constrained scenarios. The specific implementation method for this proposal is as follows: ・Instead of performing long-term training with a single optimizer using all layers of LoRA or FFT, ・Conduct shallow learning separately using three emo variants, ・Then integrate the results using TALL-Mask-Merge. As a result, ・Without relying on lengthy training sessions ・Even in resource-constrained environments ・It is possible to obtain high-precision models approaching flat minimalist architecture 4. Conclusion: Integration of Heterogeneous Emotion-Driven Models (Emotional Ensemble) The three optimizers proposed in this study (Sens, Airy, Cats) each inspect the loss landscape based on different mathematical foundations. The “Flat Minima Synthesis via Cubic Positioning” proposed in this study integrates these learning results generated under identical conditions through mask merging (e.g., TALL-Mask-Merge). This approach enables the simultaneous acquisition of “structural stability” and “expressive refinement” that cannot be achieved by a single optimization algorithm. This is expected to become a new optimization paradigm that shifts the learning process in optimization from a temporal pursuit to a spatial, multi-faceted integration. 5. Supplementary: Trial Method for Full-Layer LoRA Integration The three models were integrated by combining their respective learning results into the original model, and this new three-model system was then merged back into the original model using TM-merge. Original Model (org) ≪= TM Integration ≪= Model S (Sens), Model A (Airy), Model C (Cats) Instead of directly integrating with LoRA alone, we integrated it into the base model and then reduced these three models back to the base model using TM-merge. FFT predicts that simply merging the three models after FFT back to the original model via TM-merge will yield equivalent results. The True Nature of Loss-Saturated Learning Progress －Reflections on a Steady Decline with Minimal Stagnation－ Loss values rarely plateau or saturate, instead consistently decreasing over time. This behavior is frequently observed, particularly when loss values continue dropping to around half the initial value during the first step—raising doubts about when convergence will ever occur. So is the learning failing? I sense something in the fact that it remains normal without any tendency toward overfitting. An intuitive understanding of this suggests the possibility that the model is learning the repair of the source model as a difference. This is merely a hypothesis, and like the creation of the flat minimas mentioned earlier, we hope it will be refined into a rigorous mathematical proof by the next generation of researchers. Furthermore, the following guarantees that “as long as the loss value has amplitude, the beat (emoPulse) will not stop.” noise_base = abs(sigma_t - trust_t) + ε_s d_base = abs(N_t - d_t) + ε_t It is precisely ε_s and ε_t that generate the non-stagnant downward behavior (prevent gradient stagnation) and produce the driving force (intrinsic kinetic energy) for exploring flat minima. This can also be said to converge when the difference in loss values disappears. This design reproducibly achieves a loss of 0.30 or lower after 10,000 steps on simplenet (FashionMNIST). In SDXL testing, training with e-pred + ZtSNR—which was achievable with the previous generation EmoNavi and its variants—can also be performed with this EmoSens and its variants. This addresses the challenges of noise resistance in FM (Flow-Matching) and sampler compatibility, while simultaneously resolving issues such as color gamut limitations, which are considered weaknesses of e-pred. Training for 300 epochs using 10 reference images completed without stagnation, and we successfully created a full-layer LoRA model showing no signs of overfitting. Future Challenges: Introduction of Adaptive Accuracy Assessment Using the 8th-Order Moment Approximation Looking ahead, we are considering the introduction of a “high-order accuracy assessment mechanism” utilizing dNR cubed (equivalent to the eighth-order moment) and other metrics. This is an attempt to utilize the eighth-order information not as a direct output of emoPulse, but as a meta-indicator for evaluating the “purity” of the current learning process. This is expected to enable the detection of overfitting signs in extremely small datasets at an even earlier stage, thereby maximizing the precision of autonomous control. References Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980. Reddi, S. J., Kale, S., & Kumar, S. (2019). On the Convergence of Adam and Beyond. ICLR. Defazio, A., & Mishchenko, K. (2023). Learning-Rate-Free Learning by D-Adaptation. ICML.) Orabona, F., & Tommasi, T. (2017). Training Deep Networks without Learning Rates Through Coin Betting. NeurIPS. Luo, L., Xiong, Y., & Liu, Y. (2019). Adaptive Gradient Methods with Dynamic Bound of Learning Rate. ICLR. Shazeer, N., & Stern, M. (2018). Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. ICML. Bernstein, J., Wang, Y. X., Azizzadenesheli, K., & Anandkumar, A. (2018). signSGD: Compressed Optimisation for Non-Convex Problems. ICML. Chen, S. B., et al. (2023). Symbolic Discovery of Optimization Algorithms. arXiv. Zeyuan Allen-Zhu. (2017). Natasha: Faster Non-Convex Optimization Than SGD. arXiv.