Spectacular failure of Lebesgue differentiation for rectangles

Let $\mathcal{R}$ be the set of rectangles in the plane and, given $f \in L^1$ let $$ f^*(x) = \sup_{x \in R \in \mathcal{R}} \frac{1}{ \lvert R \rvert} \int_R \lvert \, f \,\rvert $$ as defined in [this question](https://math.stackexchange.com/questions/641399/maximal-functions-where-weak-type-inequality-fails). You can show that the weak-type inequality fails for this operator, that is, there is no constant $A$ such that $$ m(\\{ f^* > \alpha \\}) < \frac{A}{\alpha} \lVert \, f \, \rVert_1 $$ for each integrable $f$. From Stein and Shakarchi's book on real analysis I'd like to show the following, much stronger claim: there is an integrable $f$ such that

$$ \limsup_{\text{diam}(R) \to 0} \frac{1}{|\,R\,|} \int_R \lvert \, f \, \rvert = \infty \text{ a.e.} $$ where diam, of course, is the diameter. According to the book this "should" follow from the failure of the weak type inequality, but I don't really know how to do it.

I was thinking something like the following, but it failed. It suffices to find and $f$ such that the desired conclusion holds only on a set of positive measure, for we can translate around and get the desired effect. So I wanted to show that if the $\limsup$ expression is finite everywhere, the weak-type inequality holds, for that specific function. That would get me what I want. However I got nowhere with finding a function for which the weak-type inequality fails, sadly.

Edit: Whoops, this is actually very hard. In its most general form it relies on a result found on page 441 of Stein's book Harmonic Analysis: Real-Variable Methods, Orthogonality, and Oscillatory Integrals. Hopefully this specific case can be had more easily, though?

Thanks for referring to Stein's proof. I did struggle with (15), until I realized that $|E_k|$ is bounded by the size of $B$ so $||g'_k|| \rightarrow 0$ as $k\rightarrow\infty$.

Have you studied Stein's proof? It is quite straightforward and most of the work is in making formal your intuition that you can find a contradiction on a compact set and translate it around.


Is there an open subset $A$ of $[0,1]^2$ with measure $>\frac{1}{100}$ that satisfies this property?

Can we find for any given $\varepsilon>0$ an open subset $A\subseteq[0,1]^2$ with measure $>\frac{1}{100}$ such that, for any smooth curve $\gamma:[0,1]\to\mathbb{R}^2$ of length $1$, the set $\gamma+A=\\{\gamma(t)+a;t\in[0,1],a\in A\\}$ does not contain any balls of radius $\varepsilon$?

I wouldn't mind changing the $\frac{1}{100}$ for any other positive constant. Also, I ask about smooth curves but it may make more sense to consider in general $1$-Lipschitz functions $\gamma:[0,1]\to[0,1]^2$.

For context, a positive answer to this question could be useful for [this other question](https://mathoverflow.net/questions/420897/what-is-the-smallest-size-of-a-shape-in-which-all-fixed-n-polyominos-can-fit/421175#421175). But the question is also interesting in itself, of course.

Now posted to MO, mathoverflow.net/questions/422493/…

Yes, I was asking having in mind the negative answer (baseless belief as well!)

Oh and if you mean that if the curves $\gamma$ have length $10$ the answer is negative then yes, I am interested (for some reason I think that the answer of these problems is going to be positive but it's a baseless belief)

The case of $\gamma$ having length $10$ would imply the question as above, because any smooth curve of length $1$ is a subset of a smooth curve of length $10$. But any ideas of how to solve problems similar to the one above are welcome. For example I would also be interested in the case of the curves $\gamma$ having length $\leq\frac{1}{2}$

Would you be interested in the case when $\gamma$ has length, say, 10?

Btw, I am not sure why the "multivariable calculus" tag is more appropiate than the "recreational mathematics" one on this question. This is not part of any mathematics curriculum that I know of

Thanks, that's what I missed: the restriction on length.

@mihaild I don't understand very well, your curve $\gamma$ is $[0,1]\to\mathbb{R}$, not $[0,1]\to\mathbb{R}^2$, and I don't think it has length $1$

If $A$ is open, it contains an open ball of radius $x$, now if we take $\gamma(t) = \sin\frac{20 \pi t}{x}$, $\gamma +A$ fills unit square  (as moving even just this ball along $\gamma$ fills it). I guess something is wrong with this reasoning, but I can't find what.


Closed form expression for $\sqrt{4+\sqrt[3]{4+\sqrt[4]{4+\dots}}}$

Since $e=\sum_{n=0}^\infty\frac{1}{n!}=1+1+\frac12(1+\frac13(1+\frac14(1+\dots)))$, we have

$$4^{e-2}=\sqrt{4\cdot\sqrt[3]{4\cdot\sqrt[4]{4\cdots}}}$$

Is there however a nice way to express the radical in the title too?

It might be interesting to you that by taking $4$ out of the first root, we obtain another form of this radical: $$2\sqrt{1+\sqrt[3]{\frac{1}{2^4}+\sqrt[4]{\frac{1}{2^{22}}+\dots\sqrt[k]{\frac{1}{2^{k!-2}}+\dots}}}} \approx 2.401615526...$$ You just have to carefully take $1/4$ under each root, moving further and further

I have to agree, you probably won't find an answer to this.  Noting that the power of the radicals changes (or if the values inside change) makes me want to say there is no closed form.

@BarryCipra Well that's unfortunate... these things aren't perfect. On the other hand, a constant like $4^e$ seems pretty rare, I don't think I've ever gotten anything quite like it when evaluating an integral, say. But it does seem weird that it wouldn't be detected.

@BrunoJoyal, I agree it's a good idea, but with one caveat:  I ran $4^{e-2}\approx2.70675377667$ through an ISC, just as a "sanity check," and came up empty.

I would try to evaluate it to high precision and run the resulting approximation through a reverse symbolic calculator to see if anything comes up.


Evaluating a Polynomic-Trigonometric-Hyperbolic Integral

Within [this AoPS thread](https://artofproblemsolving.com/community/c7h1746945p11375644) it is asked to evaluate the following integral

> $$\mathfrak I~=~\int_0^\infty \frac{x\sin x}{\cos x+\cosh^2 x}\mathrm dx\tag1$$

In order to be precise there is also a possible closed-form conjectured which is given by

> $$\mathfrak I~=~G-\frac12\tag2$$

But as it is pointed out within the linked thread this seems to be only a reasonable approximation off after the $5$th decimal digit.

I have to admit that it is highly improbable that there exists a nice looking closed-form for $(1)$ since the integrand involves polynomials, trigonometric aswell as hyperbolic functions. I am not even sure how to get started, i.e. which substitution to choose or which technique at all to start with.

A related, but perhaps more handable integral, would be the following

> $$\mathfrak J~=~\int_0^\infty \frac{\sin x}{\cos x+\cosh^2 x}\mathrm dx\tag{1$'$}$$

Out of experience I could imagine that $(1')$ may has a closed-form in terms of known constants $($or series$)$ since it only contains the two closely connected trigonometric and hyperbolic functions.

> Is it in fact possible to deduce a closed-form for $(1)$ and $(1')$? For myself I cannot offer an approach since everything I tried was not helpful at all hence I was not even able to perform one or two steps in order to simplify the given integrals. I would be glad to see a full solution or even attempts in evaluating $(1)$ and $(1')$ since I have no idea how to deal with such integrands.

Thanks in advance!

**EDIT**

Out of pure chance I just stumbled upon [a related MSE question](https://math.stackexchange.com/questions/1798467/why-does-the-hard-looking-integral-int-0-infty-fracx-sin2x-coshx/1798559#1798559) dealing with the integral

$$\int_0^\infty\frac{x\sin^2x}{\cosh x+\cos x}\mathrm dx=1$$

Which on the other hand motivates me to believe that there may be a closed-form for $(1)$.

@GEdgar I have problems with applying the residue theorem for complicated integrals correctly. Are you here able to do so?

@Zacky I am aware of this list but has not thought about searching within for the given integral. As you already mentioned this list is unreliable but cotains some "good" approximations but way to many completely wrong values.

The integral also appears within this list: en.wikiversity.org/wiki/User:Integrals123 (I typed in ctrl + F cosh and scrolled down till the $54$-th integral), which unfortunately has many wrong answers since it was done with Inverse Symbolic calculator and that result kinda played us. But yes a closed form would be interesting even though it's not $G-\frac12$.

Note.  Since the integrand is even, we ave $$2 \mathfrak I~=~\int_{-\infty}^\infty \frac{x\sin x}{\cos x+\cosh^2 x}\mathrm dx$$ and there is a chance this could be done using a contour in the complex plane.


How to generalize Reshetnikov's $\arg\,B\left(\frac{-5+8\,\sqrt{-11}}{27};\,\frac12,\frac13\right)=\frac\pi3$?

Given the [argument](https://en.wikipedia.org/wiki/Argument_\(complex_analysis\)) $\arg(z)$, we can observe that for $k=1,2,3$, $$\arg z_1 = \frac{k\,\pi}3, \quad z_1 = \left(\tfrac{1+\sqrt{-3}}{2}\right)^k\qquad\tag1$$ $$\qquad \arg z_2=\frac{k\,\pi}3, \quad z_2 = \left( B\Big(\color{blue}{\tfrac{-5+8\,\sqrt{-11}}{27}};\,\tfrac12,\tfrac13\Big)\right)^k \tag2$$ and _incomplete beta function_ $B(z;a,b)$. [The second](https://math.stackexchange.com/questions/687567/closed-form-for-2f-1-left-frac12-frac23-frac32-frac8-sqrt11-i-5) with $k=1$ is by V. Reshetnikov.

But that cannot be an isolated result. Note that $z_1^3=-1$ while the complex number $z_2$ has a _real_ cube $z_2^3 \approx 6.1319080509$ with no known closed-form.

Reshetnikov states (without giving details) that $(2)$ is equivalent to, $$B\left(\frac19;\,\frac16,\frac13\right) = \frac12\,B\left(\frac16,\frac13\right) =\frac{1}{2\sqrt{\pi}}\,\Gamma\left(\frac16\right)\Gamma\left(\frac13\right)\tag3$$ Thus, $(3)$ is related to, $$I\left(\color{blue}{\frac19};\,\frac16,\frac13\right)=\frac12 \tag4$$ with _regularized beta function_ $I(z;a,b)$. The equation $$I\left(z;\,\frac16,\frac13\right)=\frac1n \tag5$$ to find some algebraic $z$ for a given integer $n$ is quite easy to solve. For example, the smallest real root of, $$-1 + 99 z - 243 z^2 + 81 z^3= 0\tag6$$ will yield $n=3$. More specifically, $$I\left(\frac19\Big(1-4\sin\big(\tfrac{\pi}{18}\big)\Big)^2;\,\frac16,\frac13 \right)=\frac13\quad\quad\tag7$$ $$\quad I\left(\frac13\Big(1-\frac{\sqrt2}{\sqrt[4]3}\Big)^2;\,\frac16,\frac13\right) =\frac14\quad\tag8$$ and so on.

> **Q:** But how do we get from $\color{blue}{\frac{-5+8\,\sqrt{-11}}{27}}$ of $(2)$ to $\color{blue}{\frac19}$ of $(4)$? (I assume a hypergeometric transformation is involved?)

If the answer can be found, then perhaps we can use an expression $P(z)$ of a root $z$ of $(6)$ to find another, $$\arg z_3=\frac{\beta\,\pi}3,\quad z_3=B\Big( P(z);\,\tfrac12,\tfrac13\Big)\quad\tag7$$ where $\beta$ is a rational/algebraic number?

The exponent in $(2)$ should be inside $\arg$, no?


Covering number/Metric Entropy of the unit ball with respect to Mahalanobis distance

Let $B$ denote the unit ball on $\mathbb{R}^d$ and $N(\epsilon, B, d)$ be the cardinality of the smallest $\epsilon$-cover of $B$. An epsilon cover is a set $T \subset B$ such that for any $x \in B$, there is a $t \in T$ with $d(t,x) \le \epsilon$. See for example [here](https://www.stat.berkeley.edu/%7Ebartlett/courses/2013spring-stat210b/notes/12notes.pdf). $N$ is referred to as the covering number, and $\log N$ is the Metric entropy.

Consider the following result: let $\|\cdot\|$ be a norm on $\mathbb{R}^d$ then $$ \frac{1}{\epsilon^d} \le N(\epsilon, B, \|\cdot\|) \le \left (1+\frac{2}{\epsilon} \right)^d. $$

I would like to know if there are bounds on the covering numbers that are dimension free when we choose the metric to be the Mahalanobis distance $d_S(x,y) = \|S^{-1/2}(x-y)\|_2$ for some positive definite covariance matrix $S$. Are there results along the lines of:

$$ \frac{1}{\epsilon^{f(S)}} \le N(\epsilon, B, d_S) \le \left (1+\frac{2}{\epsilon} \right)^{f(S)}. $$ where $f(S)$ is some quantity depending on $S$? An example I have in mind is when $S$ is diagonal with quickly decaying diagonal elements, e.g. $S_{ii} = i^{-2}$.


Expectation of maximum of minimums of permutations

Assume $n$ random permutations $\pi_1,\pi_2,\ldots,\pi_n: \lbrace 1,2,\ldots,m \rbrace \rightarrow \lbrace 1,2,\ldots,m \rbrace$. Let $X_i = \min(\pi_1(i),\pi_2(i),\ldots,\pi_n(i))$ and $Y = \max(X_1, X_2, \ldots, X_m)$. What is the expectation of $Y$, $E(Y)$, as function of $n$? An upper bound approximation for $E(Y)$ would also be very helpful.

Obviously, we have $E(Y)=m$ for $n=1$ and $E(Y)=1$ for $n\rightarrow\infty$.

I know that the distribution of $X_i$ is given by $P(X_i\leq k) = 1 - \left(1-\frac{k}{m}\right)^n$. However, since $X_1,X_2,\ldots,X_m$ are not independent, it is not possible to get the distribution of $Y$ by $P(Y \leq k) = P(X_1 \leq k \wedge X_2 \leq k \wedge \ldots \wedge X_m\leq k) \neq P(X_1 \leq k) \cdot P(X_2 \leq k) \cdot\ldots \cdot P(X_m \leq k)$.


Conditional expectation continuous in the conditioning argument?

Let $X$ and $Y$ be random vectors defined on a common probability space. $X$ takes values in a finite-dimensional space $\mathcal{X} \subset \mathbb{R}^p$, while $Y$ takes values in $\mathbb{R}$. The conditional expectation $E(Y \mid X)$ is then a random variable that is uniquely defined up to null sets.
I am seeking a set of sufficient conditions on the joint distribution of $(X,Y)$ for the following statement to be true:

Given any point $x_0 \in \mathcal{X}$, there exists a function $f \colon \mathcal{X} \to \mathbb{R}$ such that (i) $f(X)$ is a version of $E(Y \mid X)$, and (ii) $f(\cdot)$ is continuous at $x_0$.

Obviously, the continuity part is the non-trivial one. By Lusin's theorem, any measurable function (such as any version of the conditional expectation function) is "nearly continuous", but this is not quite enough for me.
Ideally the sufficient conditions for the above statement would not involve restrictions on the densities or conditional densities of $X$ and $Y$. The problem that motivates this question has a complicated geometry, so it is difficult to characterize densities with respect to fixed dominating measures.
If you require more structure to the problem (but ideally the question would be answered in more generality), you may assume: $Y = g(A)$ for a continuous function $g(\cdot)$, $X = A + B$, and the random vectors $A$ and $B$ are independent. However, $A$ and $B$ may concentrate on different subspaces of $\mathbb{R}^p$, each of which is a complicated manifold.
Thank you in advance for your time!


Changing FrontEnd automatic scrolling in version 8

In _Mathematica_ versions < 8, the FrontEnd has a very intelligent behavior:

  * On evaluation, it by default automatically scrolls down the Notebook window to the last printed Output cell but also allows a user to scroll the Notebook window up by hand and then does not scroll down the window again automatically.

  * If the user wishes to restore automatic scrolling it is sufficient to scroll down the window to the last currently printed Output cell and automatic scrolling will be restored.

I feel such behavior very comfortable. But in _Mathematica_ 8 we have no such behavior by default. I found that it may be partially restored by setting

    SetOptions[$FrontEnd,
     EvaluationCompletionAction -> {"ScrollToOutput"}]

But then it is not possible to stop automatic scrolling by scrolling the window by hands. It is possible to restore the old scrolling behavior?

* * *

Through _Mathematica_ 10.4 the old scrolling behavior has not been restored.

  * Have any new options come on line to control this?

  * Is there a hook to determine scroll position that might be used for a workaround?

  * Could [`PrintTemporary`](http://reference.wolfram.com/language/ref/PrintTemporary.html) and/or [`Dynamic`](http://reference.wolfram.com/language/ref/Dynamic.html) (which is active only when visible) be used to simulate the old behavior?

  * If the old behavior is simply not achievable what is the best alternative for a similar workflow?

@Wizzerad In Mathematica version 12.0 your problem seems to be resolved.

@Wizzerad I suggest you to ask the tech support about it because (as you see) no one here seems to know how to get tiny control on the FrontEnd behavior while evaluating. If you get a solution, please post it here - it will help others.

In Mathematica 10, it does not seem possible to peacefully write some code while it is evaluating. It keeps jumping back and forth between the code line I am writing and the end of the input part of the section that is being evaluated. Is there a way to normally work on the code while Mathematica 10 is evaluating part of the notebook?

In version 10.0 I have that, by default, the front end always auto-scrolls to view new output, regardless of previous hand scrolls by the user.  I take it from this question that I may disable all auto-scrolling by setting EvaluationCompletionAction to something else.  Is that correct?  Are there any other options as of version 10?

In March 2015 I send a suggestion to WRI tech support (CASE:2679254) with a link to this thread. They replied that it was forwarded to "the appropriate people in our development group", so I think we at least have a hope. I have nothing against the new section in my question.

Alexey, I added a new section to this question probing for alternatives and workarounds, and noting that the behavior has not been restored through 10.1 and is presumably not going to be.  I hope you approve.

@shrx I mean at least Mathematica versions 5, 6 and 7 (I worked with).

@Alexey Popkov It is unclear, do you mean: "In Mathematica versions < 8..."?

@Sjoerd I agree that editing during evaluation was not always stable (of course, it does not mean that this is bad idea itself). The main advantage of the old behavior is the ability to switch between inspecting of already printed outputs and automatic scrolling.

Well, it happened to me quite a lot that I got drawn to the bottom of the print area at times I didn't want to be there and when I succeeded staying above the scroll area it wasn't very stable either.

@Sjoerd With the old behavior we could not only edit the input during evaluation (as we can by default now) but also inspect already printed outputs from the running evaluation (by scrolling the page). And all of this without loosing automatic scrolling when we wish! I see no advantages in loosing this.

Actually, I found the old behaviour rather annoying. Quite distracting when you were editing and running evaluations at the same time.


Is it possible to regain Mathematica 5.2's palette input focus behaviour with version 8.0?

Between Mathematica 5.2 and later versions, there has been a change in determining which notebook gets the palette input focus, which leads to quite unfortunate behaviour if you use focus-follows-mouse as I do.

For those who don't know focus-follows-mouse: This mode gives the keyboard input focus to the window which currently contains the mouse pointer. Now consider the following situations: There are two notebooks and a palette open (one of the notebooks might be the help window, which under Mathematica is basically just a notebook). Let's say, the first notebook is on the left, the second one is in the middle, and the palette is on the right. Let's also say that I'm working on the left notebook, and now want to use the palette. Therefore I move the mouse to the palette and press the button I want.

In Mathematica 5.2 this had exactly the intended effect: Mathematica remembered that I last interacted with the left notebook, and inserted the input generated by the palette there. However in Mathematica 8.0 the input is instead inserted in the _right_ notebook.

Of course it's not hard to understand how this happens (and indeed, the problem is not restricted to Mathematica, but occurs also e.g. with Gimp; however the fact that Mathematica 5.2 got it right gives me hope that it's just some option I have to toggle or some initialization I have to edit to regain the behaviour). When I move the mouse pointer from the left notebook to the palette, it of course crosses the right notebook which is in between, and therefore focus-on-mouse gives the keyboard input focus to that window. Of course it loses the input focus again when I move on to the palette (which ultimately gets the input focus), but newer Mathematica versions obviously record which window had input focus last and use _that_ to determine which notebook the palette should act on.

Therefore my question is: Is there some option I can set to recover the old Mathematica-5.2 behaviour? Or if not, is there a way to get it with some Mathematica programming (which I then could put into an init file)?

**Edit**

As I said in a comment to an answer that was deleted in the mean time, I've found out that Mathematica 7 still behaves in the way I want. So the change was obviously introduced in between 7 and 8. I'm adding that here so that the information is visible to those who cannot see deleted answers (and in case the deleted answer including comments might eventually go away completely).

Also note that (as I also wrote in that comment) `SelectedNotebook[]` changes the same way (i.e. follows focus unconditionally in version 8, but changes only on interaction in v7).

I'd like to add a related difference between 5.2 and 8 which annoyes me a bit: select (highlight) any Keyword, say "Solve", in the code, then press F1 to get help. The window with the help system opens and displayes the explanation of that keyword. Now continue to work in your notebook hereby moving the cursor to another position. Now you wish to come back to the help for our keyword "Solve". Pressing F1 in 5.2 leads you where you want, 8 doesn't (but to the character at your pursor position). So some automatic focusing mechanism in 8 leads to an unwanted (from my point of view) effect.

AFIK, OS X doesn't have focus-follows-mouse. Keyboard focus remains with the active window even if the mouse focus moves to, say, a floating palette.

Ah, and note that those previous versions show the good behaviour runnning on the very same computer, under the very same operating system. Oh, and I just noted that I made a mistake in my description of focus-follows-mouse; fixed now.

Do you have Focus-follows-mouse on OS X (I never worked with it, so I can't tell, but e.g. Windows doesn't have it)? Because the problem description only makes sense with Focus-follows-mouse. The point is, previous versions of Mathematica contain code that creates the good behaviour (and that code must have been there on purpose, because a naive implementation will give the exact behaviour I see with 8.0.0.0), and therefore I think that code might also be in 8.0.0.0, but just not activated for some reason (I hope it's still there). Of course it could also be a bug which was fixed in 8.0.4.

I can not reproduce this problem with Mathematica 8.0.4 running under OS X 10.6.8 on a Mac. Are you sure this isn't a problem with your OS?


What is the complete list of ExportPacket formats?

`ExportPacket` is the most fundamental way we can turn `Box` forms into string data in other formats via the FE.

For example, a classic way this works is:

    First@FrontEndExecute@
      ExportPacket[Cell@BoxData@ToBoxes@Hyperlink["asdasd", "asdasd"],
       "PlainText"]

    "asdasd"

Or

    First@FrontEndExecute@
      ExportPacket[Cell@BoxData@ToBoxes@Hyperlink["asdasd", "asdasd"],
       "InputText"]

    "Hyperlink[\"asdasd\", {\"asdasd\", None}]"

It's enormously useful, but obviously entirely undocumented.

Now I'm interested in figuring out what other options it can take. Currently I've scraped up the following formats:

    {
     "PostScript", "InputForm", "GIF",
     "BoundingBox", "NotebookString",
     "EnhancedMetafile", "Metafile", "MGF",
     "CDFNotebookString", "PDF",
     "PICT", "BitmapPacket",
     "InputText", "PlainText", "DefaultText" (* thanks to John Fultz for this one *)
     }

But it can take more [per John Fultz](https://mathematica.stackexchange.com/questions/1319/how-do-i-extract-the-contents-of-a-selected-cell-as-plain-text/1411#comment3519_1411)

So what's the complete list of formats? As a real long shot, what're the other options it can take?

* * *

### Scraping notes

For those interested, I just scraped those formats from the `DownValues` of various `Export` functions:

    System`ConvertersDump`installSource[
      DeleteMissing@
       Flatten@Lookup[Quiet[getFormatExportData /@ $ExportFormats],
         "Sources"],
      Automatic
      ];
    epfuncs =
      Quiet@
       ToExpression[DeleteDuplicates@Join[Names["*`*"], Names["*`*`*"]],
        StandardForm,
        Function[Null,
         If[FreeQ[DownValues[#], ExportPacket | FrontEnd`ExportPacket],
          Nothing,
          #
          ],
         HoldAllComplete
         ]
        ];
    epdats =
      AssociationMap[
       Cases[
         DownValues@#,
         p : (ExportPacket | FrontEnd`ExportPacket)[_, type_, opts___] :>
          {
           "Type" -> type,
           "Opts" -> {opts}
           },
         \[Infinity],
         Heads -> True
         ] &,
       epfuncs
       ];
    Cases[Flatten@Values@epdats, ("Type" -> t_) :> t] // DeleteDuplicates

In 12.3 it looks like Rasterize got a performance boost through "ImageObjectPacket". I don't know if this format is in older versions though.


Creating compiled search TRIE file for argument string completion

I'd like to generate a compiled search TRIE file (like those found in `$InstallationDirectory\SystemFiles\FrontEnd\SystemResources\FunctionalFrequency\`) to help implement the auto-completion feature on user-defined functions.

As described in [File-name completion for custom functions](https://mathematica.stackexchange.com/q/56984), I could directly add a list of strings into the specialArgFunctions.tr file. However, my lists of arguments is long and likely to change during development, so I'd prefer to point to a TRIE file that I can update as needed.

How can I create a compiled TRIE file? Can this be done in Mathematica or Workbench? Is the TRIE file extension structure documented anywhere?

* * *

> b3m2a1 comment:
>
> These files are here in v11:
>
>
>     FileNames["*.trie",
>       PacletFind["AutoCompletionData"][[1]]["Location"],
>       Infinity
>      ]
>
>
> Similarly in 11 is [`CA`CADumpTriePacket`](https://mathematica.stackexchange.com/a/13452/38205) which seems to extract the properties from a trie file, but not the trie format itself.

@Szabolcs, thanks for the suggestion. I will try it out and let you know if I make any progress. Thanks again!

You may be interested in this: mathematica.stackexchange.com/a/129910/12  It shows how to add completion without editing specialArgFunctions.tr.  What I don't know how to do is enable completion of non-string arguments, as all examples that do this use .trie files and I don't know how to create such files.

@DanielLichtblau, Thanks for the link to 441, but I am not asking about general practices for implementing a trie data structure in Mathematica. I am looking for the specific TRIE file structure used by Mathematica's FrontEnd.


How to modify NDSolve`StateData without crashing the kernel?

Probably a hard question, but it's better to cry out loud.

[Reminded by Chris K](https://chat.stackexchange.com/transcript/message/51136457#51136457), I noticed my [`fix`](https://mathematica.stackexchange.com/a/129193/1871) function has been broken since _v11.3_. After some checking, I found `NDSolve`StateData[…]`, which is not an atom in _v9_ and becomes an atom now, can no longer be modified with pattern matching.

To keep `fix` up to date, I tried the solution in this [excellent post](https://mathematica.stackexchange.com/q/96225/1871) about modifying data inside atom, sadly the modified `NDSolve`StateData` crashes the kernel in the `NDSolve`ProcessSolutions` stage **even in _v9_** and I can't figure out a workaround.

The following is a simplified example attempting to modify the difference order to `2`. The last line crashes the kernel of course so please save your work before testing.

    tmax = 10; lb = 0; rb = 5;
    system = With[{u = u[t, x]}, {D[u, t] == D[u, x, x], u == 0 /. t -> 0,
                                  u == Sin[t] /. x -> lb, u == 0 /. x -> rb}];

    {state} = NDSolve`ProcessEquations[system, u, {t, 0, tmax}, {x, lb, rb}];

    teststate =
      state /. a_NDSolve`FiniteDifferenceDerivativeFunction :>
        RuleCondition@
         NDSolve`FiniteDifferenceDerivative[a@"DerivativeOrder", a@"Coordinates",
          "DifferenceOrder" -> 2, PeriodicInterpolation -> a@"PeriodicInterpolation"];

    Head[#]["DifferenceOrder"] & /@
     teststate["NumericalFunction"]["FunctionExpression"][[2, 1]]
    (* Should give {{2}} if the replacement succeeds. *)

    (*Failed attempt: *)
    ml = LinkCreate[LinkMode -> Loopback];
    LinkWrite[ml, With[{e = state}, Hold[e]]];
    holdstate = LinkRead[ml];

    newstate = holdstate /.
       a_NDSolve`FiniteDifferenceDerivativeFunction :>
        RuleCondition@
         NDSolve`FiniteDifferenceDerivative[a@"DerivativeOrder", a@"Coordinates",
          "DifferenceOrder" -> 2, PeriodicInterpolation -> a@"PeriodicInterpolation"] //
      ReleaseHold

    NDSolve`Iterate[newstate, tmax]

    (*Warning: the following line crashes the kernel.*)
    u /. NDSolve`ProcessSolutions@newstate

Any way to avoid the crashing and fix my `fix` function?

* * *

A bit of spelunking shows this seems to be (at least partly) related to `Internal`Bag`. A minimal example:

    bag = Internal`Bag[]
    ml = LinkCreate[LinkMode -> Loopback];
    LinkWrite[ml, With[{e = bag}, Hold[e]]]
    holdbag = LinkRead[ml]
    LinkClose[ml]
    ReleaseHold[holdbag] === bag
    (* False *)

I think the relation to Bag is only really in the fact that both are mutable object stored at the C++ level


Customized StepSize control in unconstrained gradient-descent optimization

I have a type of minimization problem I frequently solve that involves warping a large 2D triangle-mesh (~25,000 vertices) to fit a model. In the mesh, vertices carry the empirical measurements (each of which has a predicted position in the model, which is a continuous field). The potential/objective function for the system is something like this:

    potentialFn[{X0_List, triangles_List}, Xideal_List, X_List] := With[
      {elens0 = EdgeLengths[X0, triangles],
       elens  = EdgeLengths[X, triangles],
       angs0  = AngleValues[X0, triangles],
       angs   = AngleValues[X, triangles]},
      Plus[
        Total[(elens0 - elens)^2],
        Total[(angs0 - angs)^2],
        Total@Exp[-(X - Xideal)^2],
        0.25*Total[1 + (angs0/angs)^2 - 2*(angs0/angs))]]];

The idea is that the potential is equal to the sum of the squared deviations in the distances between vertex neighbors (the edges) plus the sum of the squared deviation in the angles of the mesh, plus the goal function, which is 0 when the vertices are perfectly aligned with the model and otherwise monotonically increasing with distance from an ideal fit, and finally a term that is designed to make the potential approach infinity as any triangle approaches inversion; i.e., the last term (which is similar to models of the van der Waals potential) constrains the potential such that triangle ABC would have to cross a singularity in order to becomes triangle ACB (in terms of a counterclockwise ordering of vertices). Additionally I have well-tested analytical functions that calculate the gradients (but nothing for the Hessians). I've confirmed that the gradient descent works correctly by, among other things, running a gradient descent search with a very small step-size for a very long time.

Most of those details are irrelevant to my question, however. What is important is that I can, for any set of vertex coordinates that are valid (i.e., no triangles have been inverted), calculate a maximum steps-size `S` such that for any actual step size `s` (`0 < s < S`) the resulting vertex coordinates will also be valid; so long as my step sizes always follow this rule, I can guarantee that no triangles will invert. The problem I have is that there doesn't seem to be a way for me to provide this information to the Mathematica line-search algorithm in functions like `FindMinimum`.

Initially, I thought that something like this would be the solution:

    FindMinimum[
      potentialFn[{X0, triangles}, X],
      {X, X0},
      Gradient :> potentialGradient[{X0, triangles}, X],
      Method -> {
        "ConjugateGradient",
        "StepControl" -> {
          "LineSearch",
          "MaxRelativeStepSize" :> CalculateMaxRelativeStepSize[{X0, triangles}, X]}}]

This, however, gives me an error (`FindMinimum::lsmrss`, with message "-- Message Text not found -- `(CalculateMaxRelativeStepSize[{X0, triangles}, X])`") that I can only assume is due to FindMinimum's inability to interpret the delayed-rule. I've spent a lot of time looking through the Mathematica documentation on conjugate-gradient and related unconstrained search and have found nothing that indicates that I can actually control the step-size aside from setting a permanent step-size length relative to the norm of the total gradient. That is fairly useless in this kind of case, unfortunately --- I can use it, but it results in a very slow search.

My question is this: are there existing (undocumented?) methods for providing Mathematica's line-search method with a way to calculate a maximum gradient step-size?

Note: I realize I haven't provided a minimal working example of this problem. I can do so, but this would be quite an undertaking as there is a lot of context around the specific problem instances --- if anyone believes that they can help me with this kind of optimization given an example, I will work on this, but I'd appreciate some indication that the work of translating the problem into a compact instance won't be for naught before I do it.

Might an appropriate value for "RestartThreshold" be helpful?  See Nonlinear Conjugate Gradient Methods.

I interpreted you last paragraph differently.  Sorry.  In any case, I would at least take a serious look at your problem, if I could run your code.

@bbgodfrey Thanks for pointing out the minor syntax error, but the note at the end of the original post was included precisely so that people would not try to copy/paste the code then post comments like that.

You have one too many ) in the last line of potentialFn.  Also, potentialGradient and CalculateMaxRelativeStepSize are undefined.  EdgeLengths and AngleValues also appear to be undefined or misused.

I should also note that the approach in the above comment quietly crashes the kernel as often as not somewhere in the middle of the optimization.

I should have noted that the best solution I've found is to use the StepMonitor argument; whenever the previous step invalidates a triangle, I use Throw/Catch to exit the minimization and restart it with a smaller "MaxRelativeStepSize". It seems clunky and is definitely not optimal, but it works for now.


How can autocomplete entries be added for DownValues and Properties?

I'd like to assign `DownValues` to a symbol like this

    x["firstvalue"] = 1;
    x["secondvalue"] = 2;

And then have the `DownValue` keys `"firstvalue"` and `"secondvalue"` offered in an autocomplete list once I have entered `x[`.

Another useful application for this kind of modification is to have the `"Properties"` of an object like a `FittedModel` added to the autocomplete list.

It is not always easy to remember `"SinglePredictionConfidenceIntervalTableEntries"`, which is one of the 64 properties of `FittedModel`.

Can this be done? What controls the entries in the _autocomplete "engine"_?

### Addendum

[This amazing answer](https://mathematica.stackexchange.com/a/16710/92) shows a way of altering the behavior of autocomplete. I do not yet understand how it works.

There is not much hope you achieve this with version 9.0.1. My answer you were referring to only works for 9.0. since back then the autocompletion was only partially done by the frontend. Nowadays, I don't see a way to interfere (let's call it improve) with the autocompletion happening in the Mathematica front end. In my latest project this would be possible to implement, but it's not the front end.

Well, that is far and away superior to my ad hoc solution. I may have to expropriate it ... :)

Ah for this case I've added the Keys function from this answer to my toolchain.

Not on thisFit, obviously. But, DownValues[x][[All,1,1,1]] is stymied by fcn[] := . So, I've been using Case, but you have to get the pattern correct.

@rcollyer, I don't understand your question. I can always call thisFit["Properties"] to find the properties of the fit or DownValues[x][[All,1,1,1]] to get the keys of x. You have improvements upon these methods?

I can't answer the autocomplete question, but do you want a mechanism for automatically updating the properties list?


Different timing with large array assignment

Consider the following example

    n = 10^8;
    AbsoluteTiming[
     A = ConstantArray[1., n]; // anyFunc;
     A[[2 ;; ;; 2]] = -1.;
    ]

> {0.388108, Null}

    n = 10^8;
    AbsoluteTiming[
     B = ConstantArray[1., n];
     B[[2 ;; ;; 2]] = -1.;
    ]

> {0.738504, Null}

    A == B

> True

Why is the first approach faster? The only difference is, there is an arbitrary function outside the first one. I am using Mathematica 12 on windows 10 64 bit.

This $HistoryLength = 0 can be relevant in wolframscript, cause there you cannot have access to history even though is set to infinity by default. It would make sense to set it to zero if it gives performance increase.

@HenrikSchumacher: I have no idea, and the days of being motivated to dig way under the weighted covers to find out have faded. I found this and similar behaviors years ago, as did others, but the how/why of this particular case... beats me. Perhaps a Wolfram engineer might chime in.

@HenrikSchumacher I'd bet that the history tracking of the kernel keeps a reference to B but not A that causes the tensor to be copied when some elements are changed to -1.  That the tracking would be turned off when one sets $HistoryLength = 0 makes sense; but why a reference to B exists but not to A -- or why a reference to B exists at all -- I cannot explain.

@ciao Wow, that does help, but I wonder why it does so. Can you explain that. It is not like DownValues[Out] were totally cluttered...


Simple argument for unexpected behavior in SUSY model

Consider a supersymmetric theory with 3 chiral superfields, $X, \Phi_1$ and $\Phi_2,$ with canonical Kahler potential and superpotential $$ W= \frac12 h_1 X\Phi_1^2 +\frac12 h_2 \Phi_2\Phi_1^2 + fX.$$ One can show, by doing calculations, that (i) supersymmetry is spontaneously broken, but (ii) one-loop corrections do not lift the classical pseudo-moduli space.

QUESTION: is it possible to say (ii) without looking at the explicit form of Coleman-Weinberg potential, e.g. making some field redefinition which shows that this is not an interacting theory and it is very close to Polonyi model?

@Nicolo': Comparing your calculus, with the calculus done for O’Raifeartaigh model, maybe you can  check the step, or the steps, where the difference happens. Maybe this will give an idea for some possible "rule".

right, but this is not the O'Raifeartaigh model

This does not seem obvious .Following this paper, it we take a model similar to yours, but different, the  O’Raifeartaigh model $(7.32)$, one may show that quantum corrections lift the classical pseudo-moduli space, see Chapter $7.4$ and formula $7.51$.

Yay, a good question after so many days! I hope you get an answer...


Extended Born relativity, Nambu 3-form and ternary ($n$-ary) symmetry

**Background:** Classical Mechanics is based on the Poincare-Cartan two-form

$$\omega_2=dx\wedge dp$$

where $p=\dot{x}$. Quantum mechanics is secretly a subtle modification of this. On the other hand, the so-called Born-reciprocal relativity is based on the "phase-space"-like metric

$$ds^2=dx^2-c^2dt^2+Adp^2-BdE^2$$

and its full space-time+phase-space extension:

$$ds^2=dX^2+dP^2=dx^\mu dx_\mu+\dfrac{1}{\lambda^2}dp^\nu dp_\nu$$

where $$P=\dot{X}$$

Note: particle-wave duality is something like $ x^\mu=\dfrac{h}{p_\mu}$.

In Born's reciprocal relativity, you have the invariance group which is the _intersection_ of $SO (4 +4)$ and the ordinary symplectic group $Sp (4)$, related to the invariance under the symplectic transformations leaving the Poincaré-Cartan two-form invariant. The intersection of $SO(8)$ and $Sp(4)$ gives you, essentially, the unitary group $U (4)$, or some "cousin" closely related to the metaplectic group.

We can try to guess an extension of Born's reciprocal relativity based on higher accelerations as an interesting academical exercise (at least it is for me). To do it, you have to find a symmetry which leaves spacetime+phasespace invariant, the force-momentum-space-time extended Born space-time+phase-space interval

$ds^2=dx^2+dp^2+df^2$

with $p=\dot{x}$, $ f=\dot{p}$ in this set up. Note that it is the most simple extension, but I am also interested in the problem to enlarge it to extra derivatives, like Tug, Yank,...and n-order derivatives of position. Let me continue. This last metric looks invariant under an orthogonal group $SO (4+4+4) = SO (12)$ group (you can forget about signatures at this moment).

One also needs to have an invariant triple wedge product three-form

$$\omega_3=d X\wedge dP \wedge d F$$

something that seems to be connected with a Nambu structure and where $P=\dot{X}$ and $F=\dot{P}$ and with invariance under the (ternary) 3-ary "symplectic" transformations leaving the above 3-form invariant.

**My Question(s):** I am trying to discover some (likely nontrivial) Born-reciprocal like generalized transformations for the case of "higher-order" Born-reciprocal like relativities (I am interested in that topic for more than one reason I can not tell you here). I do know what the phase-space Born-reciprocal invariance group transformations ARE (you can see them,e.g., in this nice thesis [BornRelthesis](http://eprints.utas.edu.au/10689/2/Whole.pdf)) in the case of reciprocal relativity (as I told you above). So, my question, which comes from the original author of the extended Born-phase space relativity, **Carlos Castro Perelman in** [this paper](http://vixra.org/abs/1302.0103), and references therein, is a natural question in the context of higher-order Finsler-like extensions of Special Relativity, and it eventually would include the important issue of curved (generalized) relativistic phase-space-time. After the above preliminary stuff, the issue is:

> What is the intersection of the group $SO (12)$ with the _ternary_ group which leaves invariant the triple-wedge product
>
> $$\omega_3=d X\wedge dP \wedge d F$$

More generally, I am interested in the next problem. So the extra or bonus question is: what is the ($n$-ary?) group structure leaving invariant the ($n+1$)-form

$$ \omega_{n+1}=dx\wedge dp\wedge d\dot{p}\wedge\cdots \wedge dp^{(n-1)}$$

where there we include up to ($n-1$) derivatives of momentum in the exterior product or equivalently

$$ \omega_{n+1}=dx\wedge d\dot{x}\wedge d\ddot{x}\wedge\cdots \wedge dx^{(n)}$$

contains up to the $n$-th derivative of the position. In this case, the higher-order metric would be:

$$ds^2=dX^2+dP^2+dF^2+\ldots+dP^{(n-1)^2}=dX^2+d\dot{X}^2+d\ddot{X}^2+\ldots+dX^{(n)2}$$

This metric is invariant under $SO(4(n+1))$ symmetry (if we work in 4D spacetime), but what is the symmetry group or invariance of the above ($n+1$)-form and whose intersection with the $SO(4(n+1))$ group gives us the higher-order generalization of the $U(4)$/metaplectic invariance group of Born's reciprocal relativity in phase-space?

This knowledge should allow me (us) to find the analogue of the (nontrivial) Lorentz transformations which mix the

$X,\dot{X}=P,\ddot{X}=\dot{P}=F,\ldots$

coordinates in this enlarged Born relativity theory.

**Remark:** In the case we include no derivatives in the "generalized phase space" of position (or we don't include any momentum coordinate in the metric) we get the usual SR/GR metric. When n=1, we get phase space relativity. When $n=2$, we would obtain the first of a higher-order space-time-momentum-force generalized Born relativity. I am interested in that because one of my main research topics is generalized/enlarged/enhanced/extended theories of relativity. I firmly believe we have not exhausted the power of the relativity principle in every possible direction.

I do know what the transformation is in the case where one only has $X$ and $P$. I need help to find and work out the nontrivial transformations mixing $X,P$ and higher order derivatives...The higher-order extension of Lorentz-Born symmetry/transformation group of special/reciprocal relativity.

@lurscher A generalized exteded relativity theory, beyond the one pioneered by Castro, Born, Cainiello, and many others in other "flavors" ...And where the principles of relativity and quantum mechanics merge and get generalized, much like your work on categories and branes generalize point particles...Quite impressive and stunning! I presented my own "roadmap" towards "ultimate"(final?) relativity in Slovenia, IARD 2016...Anyway, I had not too much time to develop the ideas presented there. I hope that change in the near future.

i don't have a clue where you are going with all this, but it definitely looks interesting :-)


Is there a null incomplete spacetime which is spacelike and timelike complete?

Geodesic completeness, the fact we can make the domain of the geodesic parametrized with respect an affine parameter the whole real line, is an important concept in GR. Especially, because the lack of it signals singularities.

This raises the question if incompleteness of one type of geodesic implies incompleteness of the rest. In general this is not the case.

I've found examples of the following scenarios:

  * timelike complete, spacelike and null incomplete
  * spacelike complete, timelike and null incomplete
  * null complete, timelike and spacelike incomplete
  * timelike and null complete, spacelike incomplete
  * spacelike and null complete, timelike incomplete

Is it possible to construct a spacelike and timelike complete spacetime but null incomplete?

@BenCrowell I agree that in general singularities are discuss in other terms. Nevertheless, Penrose and Hawking theorems only imply geodesic incompleteness.

@yess: I see. Sorry, my comments were not that helpful then. I'll delete them.

@BenCrowell Yes I meant that. Sorry for the sloppy definition.

@MBN Yes, in fact, I look for all the examples but couldn't find the one Geroch mention as an open problem. However the paper that John links seems to settle the mattter.

@JohnRennie Thank you very much the paper seems very interesting

I seem to remember a paper by Geroch, where this was stated as an open problem. But the paper was probably from the 70's.

This paper claims to describe just such a spacetime, however it's behind a paywall.

@BenCrowell The term "parametrization of a geodesic" does not make any sense without being compatible with the ambient differentiable structure. I.e. the differentiable structure is fixed and the $\sim \mathbb{R}$ structure is unambiguously confirmed by finding a single $\mathbb{R}$-parametrization. It is a good definition.

[...] create the definition of a naked singularity, which is pretty intricate and doesn't just refer to what types of geodesics hit it.

Discussions I've seen of this kind of thing actually talk about the characteristics of the singularity, not the characteristics of the incomplete geodesics. The main cases of interest are spacelike singularities (e.g., a black hole's singularity) and timelike singularities (e.g., cosmological ones, or naked singularities). Googling does turn up discussion of null singularities as well. I'm not convinced that the classification in terms of the characteristics of the complete and incomplete geodesics is actually useful or consistent. If so, then it would seem to have been unnecessary to [...]


Equation of motion for cyclic model of the universe

I recently started to study the cyclic universe. I came across this [article](http://www.physics.princeton.edu/%7Esteinh/sciencecyc.pdf) [1]. My question is about the action used for describing the cyclic model: $$S = \int d^{4}x\sqrt{-g}(\frac{1}{16\pi G}R-\frac{1}{2}(\partial\phi)^{2}-V(\phi)+\beta ^{4}(\phi)(\rho _{M}+\rho _{R}))$$ where $R$ is Ricci scalar and $g$ is the metric. I solve the Euler-Lagrange equation for this action and find the equation of motion for $\phi$: $$\partial _{\mu }\frac{\partial L}{\partial (\partial _{\mu }\phi )} - \frac{\partial L}{\partial _{\mu }\phi } = 0$$ $$\Rightarrow \partial _{\mu }\left [ \frac{1}{2} \sqrt{-g} g^{\alpha \beta }\left ( \delta _{\mu }^{\alpha } \partial _{\beta }\phi + \delta _{\mu }^{\beta }\partial _{\alpha}\phi \right )\right ] = \sqrt{-g}\left ( -V_{,\phi }+4\beta ^{3} \beta _{,\phi }(\rho _{M}+\rho _{R})\right )$$ The radiation term is independent of $\phi$ so only $\rho _{M}$ enters the equation of motion. The zero component: $$3 a^{2} \dot{a} \dot{\phi } + a^{3}\ddot{\phi }=a^{3}(-V_{,\phi }+4 \beta ^{3}\beta _{,\phi }\rho _{M})$$ $$\Rightarrow\ddot{\phi }+3H\dot{\phi }= -V_{,\phi }+4 \beta ^{3}\beta _{,\phi }\rho _{M}$$ where $H$ is the Hubble parameter. But this result is different from what was written in the article. The difference is the coefficient of the last term. In addition, I have a problem finding Friedmann equations for this action (again in finding coefficients). Can anybody elaborate on the reason?

**Reference:**

[1] P.J. Steinhardt and N. Turok, _" A Cyclic Model of the Universe,"_ Science 296 (2002), available at [here](http://www.physics.princeton.edu/%7Esteinh/sciencecyc.pdf)

Seems strange. You have $2$ papers with same authors (J. Steinhardt and Neil Turok) and a different sign... Sounds like a typo for the equation ($1$) of the first paper.

I think that $\triangle\phi$ is zero and $\phi$ only depends on time @Trimok.

It is not completely clear for me what $(\partial \phi)^2$ is exactly. If it is $(\partial \phi)^2 = -(\partial_0 \phi)^2+(\vec \partial \phi)^2$, then the Euler-Lagrange equations give a $\square \phi =  \ddot \phi - \triangle\phi$, and not a $\ddot  \phi$. So, I think you need to reexpress, if possible, $\triangle\phi$ as a function of $\beta, \beta_{,\phi}$, etc...

Thanks very much, @Trimok. That explains the factor 4, but what about the sign? I read this article. In this paper the sign of the coupling term is negative. Which one is correct?

For the problem of the factor $4$, from the equation $(5)$ (fluid equation of motion), you see that there is a dependence $\rho_M \sim \hat a^{-3}\sim (a\beta(\phi))^{-3}$, so the term $\beta^4(\phi)\rho_m$ is (secretely...) proportionnal to $\beta(\phi)$


Radiative equilibrium in orbit of a black hole

According to [Life under a black sun](http://arxiv.org/pdf/1601.02897v1.pdf), Miller's planet from _Interstellar_ , with a time dilation factor of 60,000, should be heated to around 890C by blue-shifted cosmic background radiation.

How they arrive at that number, however, seems to me a little opaque.

As the article describes, there are two major effects to consider: gravitational blueshifting, and blue- and redshifts due to the planet's orbital motion.

Calculating the purely gravitational effects seems straightforward (although I admit I may still be missing something); given that radiative power is proportional to $T^4$, and power should scale linearly with the time dilation factor, the apparent CMB temperature should be $2.7K * 60,000^{1/4} = 42.26K$. Considering that a cold black hole occupies part of the sky, the equilibrium temperature of the planet should be slightly lower. That's clearly a long way from 890C!

It appears, then, that the majority of the heating must be a result of the circular motion of the planet in orbit. Now, it seems fairly obvious that getting precise answers will require numerical simulation, but it should be possible to at least get a close order-of-magnitude estimate based on a model of a planet moving at constant velocity through a background of the temperature calculated from gravitational effects alone. Unfortunately, though the article doesn't quote speeds, and I haven't been able to figure out how to calculate the relevant velocities for a planet is a low orbit around a rotating black hole.

So, can anybody help me fill in the blanks? If I start with a black hole of a given mass and angular momentum, and a planet in a stable circular orbit at some given radius, how do I get to an estimate of equilibrium temperature?

Funny how things come around. I have been struggling with the same thing. I follow where  the 890C comes from, given the illumination conditions that are deduced, but like you, struggle to immediately see where the blueshift factor of 275,000 came from. @Yuketerz to the rescue. physics.stackexchange.com/a/710922/43351

I was just about to embark on an answer but then noticed that the paper you reference is all about how to get to that answer. So you need to read it and then come back with what you don't understand. e.g. The calculation you have done is (a) incorrect and (b) not directly relevant to calculating the equilibrium temperature of the planet.

See Section 3 of this paper for calculating the velocity of circular orbits in Kerr spacetime.


Topological quantum error correcting codes which are not CSS codes

The most promising-seeming quantum error correction codes for the medium-to-long term are the topological codes, of which the toric code (and variants such as planar surface codes) and colour codes are the main examples.

As with essentially all approaches to quantum error correction, these are stabiliser codes. It also happens that both the toric code and colour codes are CSS codes: that is, they have a set of stabiliser generators which consist of products of $X$ operators or products of $Z$ operators, on subsets of the qubits. The useful properties of these codes for quantum error correction do not seem to particularly rely on this fact (the much more obviously relevant property is that the code distance of these codes scale with the size of the system and can be described by topological properties). So it may simply be a coincidence that these are CSS codes -- but at the same time, it is intriguing that the codes with the best known thresholds seem to be CSS codes.

**Question.** Are there known examples of topological stabiliser codes which are neither CSS codes, nor equivalent to a CSS code up to local unitaries? Or is there a reason why such codes could not exist?

A google search brought this up: arxiv.org/abs/1012.0859 ("Local non-CSS quantum error correcting code on a three-dimensional lattice", Isaac H. Kim)


What is this RPGA world/campaign design contest winner (ca. 2006?) I remember?

More than a decade ago, my best friend had an online subscription to various D&D materials. When he was back home visiting, he let me spend a few hours exploring this treasure trove. From what I can remember, there were fully indexed, digital versions of the current rule books. There were Q&A and discussion fora as well as online campaign settings somewhat like much older/slower play-by-mail. One of the campaign worlds I was particularly impressed with there was essentially a water world, where adventures took place on the high seas between islands and underwater in the deep.

What struck me the most, though, was the world design contest winner from an RPGA network contest that I had found in the campaign settings. As I recall, it was set on a world where the great heroes had just failed to prevent the invasion of some extra-planar entities that meant to take total control. Completely blown away by the unusual concept of the starting point, I had noted the name of this world and campaign setting but lost it over the years. My friend could not remember it and no longer had his subscription. I would very much like to relearn the name of this world and get more details about it as a campaign setting. Any assistance is much appreciated.

The subscription service could be DnD Insider? That however only existed from about 2009 onwards rpg.stackexchange.com/questions/193504/…

@KorvinStarmast It wasn’t, sadly, which is why I didn’t put an answer on this.

@KRyan Hoping your research is productive

Looking into this more, it seems that WotC went out of their way to not discuss the non-winning entries to that contest. Of the three finalists, only Eberron is known since the rules of the contest included the rights to all finalist submissions becoming property of Wizards of the Coast and the authors signing NDAs about them. We know that Rich Burlew (of Order of the Stick fame) was one and Nathan Toomey was the other, but nothing about what they wrote. Of the semi-finalists, 5/8 have been published elsewhere, and the other 3/8 are unknown. Investigating those 5.

Notably, the winner of the 2002 RPGA contest was Eberron, which is still a major campaign setting published by Wizards of the Coast. Definitely doesn’t match this description, but so far as I know, that was the only contest of the sort that Wizards held—certainly the only one that resulted in publication of the setting that won. Is it possible that this was runner-up or honorable mention or something from that contest?

Thanks for the advice, Jason_c_o. While I realised the superfluous details could be distracting, they also provided a context I thought might help.  No, the water world campaign is not the one about which I was asking (though I wouldn't mind learning its name and more about it, too).  It was the WoC subscription service with all the manuals, rulebooks, campaign settings and adventures that mentioned the invaded world setting.  I recall my friend's visit as being in late summer 2006, the time at which I saw the invaded world campaign: it had already won an RPGA world design contest.

Just to be make sure: The water-world you mention is not the world design contest winner you're looking for, correct? If so, you may want to cut some of the tangential information, it's distracting. Also, can you remember anything else? What year the contest was held (The RPGA Net ran many design contests, and took submissions for tournaments), or what book/magazine mentioned it? Is 2006 when you saw it, or when it was made?

Welcome to RPG.SE! Take the tour if you haven't already and see the help center or ask us here in the comments (use @ to ping someone) if you need more guidance. Good Luck and Happy Gaming!


Compatibility of impredicative Set and function extensionality

The Coq [FAQ](https://coq.inria.fr/faq#htoc41) says that function extensionality is consistent with predicative `Set`. It's not fully clear to me from this whether it's consistent with impredicative `Set` (or maybe the consistency is unknown in that case).

This question has been mentioned by Zimm i48 in this related Coq-Club thread (this post specifically).

If you're considering migration then cstheory.SE would be another good choice, imo.

Additionally, this question is about the theory of Coq rather than a programming issue with Coq and should probably be migrated to MathOverflow where it has more chances of being answered in details.

This sounds a bit misleading; I imagine that it is consistent with impredicative Set as well. But that's probably a better question for the Coq mailing list.


How can (<*) be implemented optimally for sequences?

The `Applicative` instance for `Data.Sequence` generally performs very well. Almost all the methods are _incrementally asymptotically optimal_ in time and space. That is, given fully forced/realized inputs, it's possible to access any _portion_ of the result in asymptotically optimal time and memory residency. There is one remaining exception: `(<*)`. I only know two ways to implement it as yet:

  1. The default implementation

         xs <* ys = liftA2 const xs ys

This implementation takes `O(|xs| * |ys|)` time and space to fully realize the result, but only `O(log(min(k, |xs|*|ys|-k)))` to access just the `k`th element of the result.

  2. A "monadic" implementation

         xs <* ys = xs >>= replicate (length ys)

This takes only `O(|xs| * log |ys|)` time and space, but it's not incremental; accessing an arbitrary element of the result requires `O(|xs| * log |ys|)` time and space.

I have long believed that it should be possible to have our cake and eat it too, but I've never been able to juggle the pieces in my mind well enough to get there. To do so appears to require a combination of ideas (but not actual code) from the implementations of `liftA2` and `replicate`. How can this be done?

* * *

Note: it surely won't be necessary to incorporate anything like the `rigidify` mechanism of `liftA2`. The `replicate`-like pieces should surely produce only the sorts of "rigid" structures we use `rigidify` to get from user-supplied trees.

### Update (April 6, 2020)

Mission accomplished! I managed to find [a way to do it](https://github.com/haskell/containers/pull/713). Unfortunately, it's a little too complicated for me to understand everything going on, and the code is ... rather opaque. I will upvote and accept a good _explanation_ of what I've written, and will also happily accept suggestions for clarity improvements and comments on GitHub.

### Update 2

Many thanks to [Li-Yao Xia](https://stackoverflow.com/users/6863749/li-yao-xia) and Bertram Felgenhauer for helping to clean up and document my draft code. It's now considerably less difficult to understand, and will appear in the next version of `containers`. It would still be nice to get an answer to close out this question.

@Li-yaoXia, something very similar goes on in the aptyMiddle function. The tricky bit here is being sure that the function constructed (when applying the size to your hypothetical version) creates a result with the required internal sharing.

@dfeuer I think I've reduced the problem to constructing a function whose type would look like this with dependent types: repeat :: Int -> a -> (exists (n :: Nat). Node^n a) (this existential is equivalent to what you get from throwing away the Digits fields in the definition of FingerTree), which I find awfully non-obvious to implement.

Aha! I think I've found a way to manage the complexity at a (very small) efficiency cost! The trick will be a stripped-down version of the type of rigid trees that, instead of having digits, just has numbers indicating how big those digits are. This is all the structural information we need, because replication can (I'm pretty sure) still work in redundant base 3! So we just need a function (or class method) to flesh out those rigid tree bits, and a slightly modified aptyMiddle to put it all together.

@moonGoose, by way of background, I'm the one who wrote the liftA2 implementation, and there are still parts of it that I don't understand myself. replicate was one of several extremely clever functions Louis Wasserman wrote; it's simpler than his tails but still pretty tricky.

@moonGoose, I hear you about trying to give more details, but I don't think it will help. This is most definitely not going to be a quick-and-easy question for anybody. Answering it will require a deep dive into Data.Sequence internals; explaining the performance analysis doesn't seem likely to reduce the amount of work or effort involved.

Is join $ replicate (length ys) <$> xs the same as 2?

Imo it would help everyone to explain a bit how you get those bounds, perhaps even a diagram of what the different implementations produce? Otherwise everyone who would like to think about the problem has to independently do the work to get to where you're starting from.


Metal crash upon adding SKSpriteNode to SKEffectNode

    > -[MTLDebugRenderCommandEncoder setScissorRect:]:2028: failed assertion (rect.x(0) + rect.width(1080))(1080) must be <= 240

I am getting this crash when adding a simple `SKSpriteNode` to a `SKEffectNode` with the following code

        SKSpriteNode *warpSprite = [SKSpriteNode spriteNodeWithImageNamed:@"art.scnassets/symbol.png"];
        SKEffectNode *entryEffectsNode = [[SKEffectNode alloc] init];
        [entryEffectsNode addChild:warpSprite];
        [self addChild:entryEffectsNode];

I have not touched these nodes anywhere else in my project, when i change the sprite the value in (must be <=value) changes within the error.

Edit: I have replaced the sprite image with a simple `spriteNodeWithColor:Size:` and the (<=value) is always twice size of the sprite. Also it should be noted that the SKScene is being used as a overlay in a SceneKit scene.

I have created a seperate SKScene with the following code, which still results in the same error.

    @implementation testScene

        -(id)initWithSize:(CGSize)size {
            if (self = [super initWithSize:size]) {

                SKSpriteNode *testSprite = [SKSpriteNode spriteNodeWithColor:[SKColor purpleColor] size:CGSizeMake(100, 100)];
                SKEffectNode *testEffect = [[SKEffectNode alloc] init];
                [testEffect addChild:testSprite];
                [self addChild:testEffect];

            }
            return self;
        }
    @end

Edit 2: I have just tested the above scene as an overlay on a default SceneKit Project and it crashes with the same error.

Edit 3: I have reproduced this using swift. Bug report summited to Apple.

What is the current status on your bug report?

Are you certain your initWithSize method is called on the main thread ? When using a SKScene as a SCNScene overlay, all SKNode operations must be run from the main thread.

Might be a bug. File one with Apple to be sure.

With this changed i still get the same error. What else would you need to know? I have just run the same code in a new project and it works as you would expect. I have replaced the sprite image with a simple spriteNodeWithColor:Size:  and the (<=value) is always twice size of the sprite.

your rect is greater than what is supported,   [SKSpriteNode spriteNodeWithImageNamed:@"art.scnassets/symbol.png"] should be  [SKSpriteNode spriteNodeWithImageNamed:@"symbol"] so that it can properly handle retina graphics,  Other than this, we would need to know more


Creating Spark Objects from JPEG and using spark_apply() on a non-translated function

Typically when one wants to use `sparklyr` on a custom function (_i.e._ [**non-translated functions) they place them within `spark_apply()`](https://sparkfromr.com/non-translated-functions-with-spark-apply.html). However, I've only encountered examples where a _single_ local data frame is either `copy_to()` or `spark_read_csv()` to a remote data source and then used `spark_apply()` on it. An example, for illustrative purposes only:

    library(sparklyr)
    sc <- spark_connect(master = "local")

    n_sim = 100
    iris_samps <- iris %>% dplyr::filter(Species == "virginica") %>%
      sapply(rep.int, times=n_sim) %>% cbind(replicate = rep(1:n_sim, each = 50)) %>%
      data.frame() %>%
      dplyr::group_by(replicate) %>%
      dplyr::sample_n(50, replace = TRUE)

    iris_samps_tbl <- copy_to(sc, iris_samps)

    iris_samps_tbl %>%
      spark_apply(function(x) {mean(x$Petal_Length)},
        group_by = "replicate") %>%
      ggplot(aes(x = result)) + geom_histogram(bins = 20) + ggtitle("Histogram of 100 Bootstrapped Means using sparklyr")

It seems like it would be therefore possible to use this on any range of non-translated functions coming from `CRAN` or `Bioconductor` packages as long as the data resides in a Spark Object.

I've came up with a specific problem for `.jpeg` images as I read that [`SparkR` can load compressed image (`.jpeg`, `.png`, _etc._) into raw image representation via `ImageIO` in Java library](https://spark.apache.org/docs/latest/ml-datasource) \- it seems possible that `sparklyr` could do this as well.

`RsimMosaic::composeMosaicFromImageRandom(inputImage, outputImage, pathToTilesLibrary)` function takes an input image and a path to tiles used to create a photo mosaic and outputs an image ([example here](https://rviews.rstudio.com/2020/02/13/photo-mosaics-in-r/)).

If this function only took one image and I knew how to turn it into spark object I might imagine the command would look like: `composeMosaicFromImageRandom(inputImage, outputImage, spark_obj)`. However, this function is taking a path to 30,000 images.

How would one create 30,000 Spark Objects from the path to these tiles (`.jpegs`) and then use this function?

If the underlying code would actually need to be modified I've used `jimhester/lookup` to provide the source code:

    function (originalImageFileName, outputImageFileName, imagesToUseInMosaic,
            useGradients = FALSE, removeTiles = TRUE, fracLibSizeThreshold = 0.7,
            repFracSize = 0.25, verbose = TRUE)
    {
            if (verbose) {
                    cat(paste("\n ------------------------------------------------ \n"))
                    cat(paste("    R Simple Mosaic composer - random version   \n"))
                    cat(paste(" ------------------------------------------------ \n\n"))
            }
            if (verbose) {
                    cat(paste("    Creating the library... \n"))
            }
            libForMosaicFull <- createLibraryIndexDataFrame(imagesToUseInMosaic,
                    saveLibraryIndex = F, useGradients = useGradients)
            libForMosaic <- libForMosaicFull
            filenameArray <- list.files(imagesToUseInMosaic, full.names = TRUE)
            originalImage <- jpeg::readJPEG(filenameArray[1])
            xTileSize <- dim(originalImage[, , 1])[1]
            yTileSize <- dim(originalImage[, , 1])[2]
            if (verbose) {
                    cat(paste("    -- Tiles in the Library : ", length(libForMosaic[,
                            1]), "\n"))
                    cat(paste("    -- Tile dimensions : ", xTileSize, " x ",
                            yTileSize, "\n"))
            }
            if (verbose) {
                    cat(paste("\n"))
                    cat(paste("    Reading the original image... \n"))
            }
            originalImage <- jpeg::readJPEG(originalImageFileName)
            xOrigImgSize <- dim(originalImage[, , 1])[1]
            yOrigImgSize <- dim(originalImage[, , 1])[2]
            if (verbose) {
                    cat(paste("    -- Original image dimensions : ", xOrigImgSize,
                            " x ", yOrigImgSize, "\n"))
                    cat(paste("    -- Output image dimensions : ", ((xOrigImgSize -
                            2) * xTileSize), " x ", ((yOrigImgSize - 2) * yTileSize),
                            "\n"))
            }
            if (verbose) {
                    cat(paste("\n"))
                    cat(paste("    Computing the mosaic... \n"))
            }
            outputImage <- array(dim = c(((xOrigImgSize - 2) * xTileSize),
                    ((yOrigImgSize - 2) * yTileSize), 3))
            removedList <- c()
            l <- 1
            pCoord <- matrix(nrow = ((xOrigImgSize - 2) * (yOrigImgSize -
                    2)), ncol = 2)
            for (i in 2:(xOrigImgSize - 1)) {
                    for (j in 2:(yOrigImgSize - 1)) {
                            pCoord[l, 1] <- i
                            pCoord[l, 2] <- j
                            l <- l + 1
                    }
            }
            npixels <- length(pCoord[, 1])
            for (i in 1:npixels) {
                    idx <- round(runif(1, 1, length(pCoord[, 1])))
                    pixelRGBandNeigArray <- computeStatisticalQuantitiesPixel(pCoord[idx,
                            1], pCoord[idx, 2], originalImage, useGradients)
                    tileFilename <- getCloseMatch(pixelRGBandNeigArray,
                            libForMosaic)
                    startI <- (pCoord[idx, 1] - 2) * xTileSize + 1
                    startJ <- (pCoord[idx, 2] - 2) * yTileSize + 1
                    outputImage[startI:(startI + xTileSize - 1), startJ:(startJ +
                            yTileSize - 1), ] <- jpeg::readJPEG(tileFilename)
                    if (removeTiles) {
                            libForMosaic <- removeTile(tileFilename, libForMosaic)
                            removedList <- c(removedList, tileFilename)
                            if (length(libForMosaic[, 1]) < (fracLibSizeThreshold *
                                    length(libForMosaicFull[, 1]))) {
                                    idxs <- runif(round(0.25 * length(libForMosaicFull[,
                                            1])), 1, length(removedList))
                                    for (ii in 1:length(idxs)) {
                                            libForMosaic <- addBackTile(removedList[idxs[ii]],
                                                    libForMosaic, libForMosaicFull)
                                    }
                                    removedList <- removedList[-idxs]
                            }
                    }
                    if (length(pCoord[, 1]) > 2) {
                            pCoord <- pCoord[-idx, ]
                    }
            }
            if (verbose) {
                    cat(paste("\n"))
                    cat(paste("    Done!\n\n"))
            }
            jpeg::writeJPEG(outputImage, outputImageFileName)
    }

_Please Note:_ My first attempt to speed up this code was 1) using `profvis` to find the bottlenecks (_i.e._ the for-loops) 2) use the `foreach` package on for-loops. This resulted in slower code which suggested I was parallelising at too low of a level. As I understand `sparklyr` is more about distributing the computing than about parallelizing it so perhaps this could work.

Have you successfully used spark_apply() on a single .jpeg? The help says that  you need to provide >An object (usually a spark_tbl) coercable to a Spark DataFrame.

I would like to use sparklyr on non-translated functions, specifically for composeMosaicFromImageRandom().

A long post; what exactly are you trying to achieve?


Detect ring/silent switch position change

I'm working on an app for which I would like to:

  1. respect the ring/silent switch when playing audio, and

  2. display an icon indicating that sound is muted when the ring/silent switch is set to silent.

Requirement 1 is easy: I'm using [`AVAudioSessionSoloAmbient`](https://developer.apple.com/library/ios/documentation/AVFoundation/Reference/AVAudioSession_ClassReference/Reference/Reference.html#//apple_ref/doc/constant_group/Audio_Session_Categories) as my app's audio session category, so that my audio session will mute itself when the ring/silent switch is off.

Requirement 2 seems considerably harder, because I need some sort of callback, notification, or KVO that will allow me to monitor the position of the switch, but Apple has made it clear that it is unwilling to offer an officially exposed way of doing this. That said, if I can find a nonintrusive way to monitor the switch's position, even one that is technically prohibited (like, say, an internal `NSNotification`), I would be willing to run it by Apple.

Further, I would prefer not to implement some of the polling solutions I've found elsewhere. See the Related Questions section for an example.

* * *

### What I've Learned (aka What Doesn't Work)

  * In iOS versions 4 and 5, at least, there was a trick that could be used to get the switch's position by [watching the route property of the current audio session](https://stackoverflow.com/questions/6901363/detecting-the-iphones-ring-silent-mute-switch-using-avaudioplayer-not-worki). Besides being deprecated by the `AVAudioSession` class, I can confirm that this trick is no longer an option. The current route, both as reported by the C functions comprising the deprecated `Audio Session` API and the current `AVAudioSession` class does not change when the ring/silent switch is toggled.

  * [`AVSystemController` is an internal class](https://github.com/nst/iOS-Runtime-Headers/blob/7ef0330f961248b9021e59e12aa3182440194817/PrivateFrameworks/Celestial.framework/AVSystemController.h) that seems to have a lot of promise. Invoking `- (BOOL)toggleActiveCategoryMuted` on the `sharedAVSystemController` does indeed mute my app's audio. Further, the shared singleton posts an `AVSystemController_SystemVolumeDidChangeNotification` notification when the system volume is changed via the volume buttons. Unfortunately, this notification is not posted in response to changes to the ring/silent switch (though [this dubiously attributed source says it should](http://cocoadev.com/AVSystemController)).

  * As far as I can tell, there are _no_ `NSNotification`s posted by _any_ object in response to ring/silent switch position changes. I came to this conclusion after adding myself as an observer to all notifications in the default center:

        [[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(handleNotification:) name:nil object:nil];

and then toggling the ring/silent switch. Nothing.

  * The `AVSystemController` class has a promising method with the signature:

        - (BOOL)getActiveCategoryMuted:(BOOL*)arg1;

However, there are two problems with this:

    1. Neither the return value nor the `BOOL` pointed to by `arg1` ever seem to change in response to toggling the ring/silent switch.
    2. Because of the method signature, this method is not (so far as I understand) a candidate for KVO.

****

  * I suspect that some object sends some other object(s) a [`GSEventRef`](https://github.com/kennytm/iphone-private-frameworks/blob/master/GraphicsServices/GSEvent.h) when the mute switch is changed, because I see the following in the declaration for event types:

        kGSEventRingerOff = 1012,
        kGSEventRingerOn = 1013,

However, I'm pretty sure I can't intercept those messages, and even if I could, that would be a bit more than "a little" intrusive.

* * *

### Why I Believe This Is Possible

Put simply: the Instagram app exhibits essentially this behavior. When you watch a video, it respects the ring/silent switch's setting, but displays an icon when the switch is off. The icon disappears and reappears so immediately after moving the switch that I assume it must be event-based, not polling.

* * *

### Related Questions

  * [This question dates back to iOS 4, and uses the methods that I mentioned in my first bullet above.](https://stackoverflow.com/questions/6901363/detecting-the-iphones-ring-silent-mute-switch-using-avaudioplayer-not-worki)

  * [This question is very similar to the one above.](https://stackoverflow.com/questions/7798891/detect-silent-mode-in-ios5?lq=1)

  * [This question is (much) more current, asking about iOS 7.](https://stackoverflow.com/questions/20992452/detect-silent-mode-in-ios-7) However, because I'm willing to accept a minimally intrusive breaking of the private-API rules, I would contend that this is a different question from my own.

  * [This _answer_ suggests using a polling method that I would strongly prefer to avoid.](https://stackoverflow.com/a/19013671/1292061)

The link above looks dead. The code above is hosted at:   github.com/moshegottlieb/SoundSwitch  (via stackoverflow.com/questions/20992452/…)

Instagram reportedly uses this solution, which is not pretty: sharkfood.com/content/Developers/content/Sound%20Switch

@PeterRobert No, I haven't. I ended up carefully implementing a solution by polling. It wasn't pretty.

Have you found any event based solution?


Azure App Service load balancing settings

ARM template for Azure App Service has setting to configure load balancing algorithm - loadBalancing. According to [documentation](https://learn.microsoft.com/en-us/azure/templates/microsoft.web/sites#siteconfig-object) it's available through SiteConfig object and can have following values: WeightedRoundRobin, LeastRequests, LeastResponseTime, WeightedTotalTraffic, RequestHash.

We performed some testing with Standard S1 app service plan with two instances. First instance was responding to all request with no delay, second instance was responding to all requests with 3 seconds delay, ARR affinity was turned off.

Test showed that all settings perform the same - after some ramp up time all requests spread evenly between two instances. It was not expected at least for LeastResponseTime, which intuitively suppose to direct more traffic to first instance (with low response time).

So the questions is, does this setting even work? And if it does, in what app service configuration it's respected?

The App Service Plan level might also play a role. In some cases, certain features or their behaviors might differ based on whether you're on a Standard, Premium, or Isolated plan. Although documentation doesn't typically specify this, service level might impact the sophistication of load balancing features.

The specific implementation of these algorithms by Azure might not exactly match their traditional definitions. For example, LeastResponseTime might require a significant difference in response times or a larger sample size to alter the routing decisions noticeably.

You could take a look at this case. The scale-out itself is the load balance.

Just a thought, have you tried to verify it from a different region. This latency tool may be helpful azurespeed.com

Azure App Service uses internal load balancer that is not exposed and not configurable directly (except misterios loadBalancing setting in ARM). So there is no way to change the weights.

For Weighted round robin, you can set the weight 5 for instance1, and weight 1 for instance2, what is the result? and you can refer to these load balancing algorithm description.

Azure app service has setting inside ARM template called "loadBalancing". Setting name and supported values suggest that it should somehow specify how requests are distributed between instances. However we were not able to see any difference using different values for "loadBalancing" setting for our test case. 50% of request was server by first instance, 50% of requests were server by second instance, for all supported setting value values: WeightedRoundRobin, LeastRequests, LeastResponseTime, WeightedTotalTraffic, RequestHash.

We used Netling to generate lot of HTTP GET requests to app service. App service was running simple ASP.NET MVC application that respond HTTP 200 to all get requests with "OK" in response body. App was configured in such a way that first instance was responding immediately (response time below 100 ms) and second instance was adding 3 second delay. We tried to simulate case when one of the servers experiencing high load with this test.

Not sure your question, could you describe what you have done in your test? or what is load balancing setting? and what do you expect in the result?


When malloc_trim is (was) called automatically from free in glibc's ptmalloc or dlmalloc?

There is [incorrectly documented](https://stackoverflow.com/questions/28612438/can-malloc-trim-release-memory-from-the-middle-of-the-heap/42273711) function `malloc_trim` in glibc malloc (ptmalloc2), added in 1995 by Doug Lea and Wolfram Gloger (dlmalloc 2.5.4).

In glibc this function can return some memory freed by application back to operating system using negative sbrk for heap trimming, `madvise(...MADV_DONTNEED)` for unused pages in the middle of the heaps (this [feature is in `malloc_trim` since 2007 - glibc 2.9](https://sourceware.org/git/?p=glibc.git;a=commit;f=malloc/malloc.c;h=68631c8eb92ff38d9da1ae34f6aa048539b199cc), but not in `systrim`), and probably not by trimming additional thread arenas (which are not sbrk-allocated by mmaped as separate heaps).

Such function can be very useful for some long-running C++ daemons, doing a lot of threads with very high number of mixed size allocations, both tiny, small, middle and large in required size. These daemons may have almost constant amount of live (allocated by malloc and not yet freed) memory, but at the same time grow in RSS (physical memory consumption) over time. But not every daemon can call this function, and not every author of such daemons know that he should call this function periodically.

Man page of `malloc_trim` by Kerrisk <http://man7.org/linux/man-pages/man3/malloc_trim.3.html> (2012) says in Notes about automatic calls to `malloc_trim` from other parts of glibc malloc:

> This function is automatically called by free(3) in certain circumstances.

(Also in <http://www.linuxjournal.com/node/6390/print> Advanced Memory Allocation, 2003 - _Automatic trimming is done inside the free() function by calling`memory_trim()`_)

This note is probably from source code of malloc/malloc.c which mentions in several places that `malloc_trim` is called automatically, probably from the `free()`:

     736  M_TRIM_THRESHOLD is the maximum amount of unused top-most memory
     737  to keep before releasing **via malloc_trim in free().**

     809  * When malloc_trim is called automatically from free(),
     810    it is used as the `pad' argument.

    2722   systrim ... is also called by the public malloc_trim routine.

But according to grep for `malloc_trim` in glibc's malloc: [http://code.metager.de/source/search?q=malloc_trim&path=%2Fgnu%2Fglibc%2Fmalloc%2F&project=gnu](http://code.metager.de/source/search?q=malloc_trim&path=%2Fgnu%2Fglibc%2Fmalloc%2F&project=gnu) there are: declaration, definition and two calls of it from `tst-trim1.c` (test, not part of malloc). Same result for whole glibc grep (additionally it was listed in abilists). Actual implementation of `malloc_trim` is `mtrim()` but it is called only from `__malloc_trim()` of malloc.c.

So the question is: When `malloc_trim` or its internal implementation (`mtrim`/`mTRIm`) is called by glibc, is it called from `free` or `malloc` or `malloc_consolidate`, or any other function?

If this call is lost in current version of glibc, was it there in any earlier version of glibc, in any version of ptmalloc2, or in original dlmalloc code (<http://g.oswego.edu/dl/html/malloc.html>)? When and why it was removed? (What is the difference between `systrim`/`sys_trim` and `malloc_trim`?)

It loos like it only runs automatically when freeing a large space: github.com/bminor/glibc/blob/…. Unfortunately the threshold for "large" is not a tunable.

A very good question. I find this method to test malloc_trim() on running apps: notes.secretsauce.net/notes/2016/04/…

dlmalloc versions: g.oswego.edu/pub/misc; glibc git github.com/bminor/glibc/blob/master/malloc/malloc.c; version of malloc in glibc with main_trim: github.com/bminor/glibc/blob/…. Example of RSS memory grow: bugs.python.org/issue11849 - message 133929


Connman without any user interaction

I'm trying to use Connman to manage the WiFi connection of my embedded system because it handles automagically any type of protection.

In interactive mode it's very simple:

  1. connmanctl
  2. agent on
  3. scan wifi
  4. services
  5. connect
  6. enter password if requested

On my system, the user enters the WiFi credentials (SSID, password) using a remote (web) application. Then I would use this information to setup connman using a script.

The goal is to avoid the user to select which type of protection is going to setup. I mean, most users just enter SSID/password but they don't know if it is a WPA-PSK or WEP connection.

I'm reading throught the documentation, but I'm not sure which is the correct approach:

  * a config file: <http://git.kernel.org/cgit/network/connman/connman.git/tree/doc/config-format.txt>

but as far as I understand I need to specify the type of the security:

> Security: The security type of the network. Possible values are 'psk' (WPA/WPA2 PSK), 'ieee8021x' (WPA EAP), 'none' and 'wep'. When not set, the default value is 'ieee8021x' if an EAP type is configured, 'psk' if a passphrase is present and 'none' otherwise.

It seems 'wep' is not handled if the field is omitted.

  * dbus-api: <http://git.kernel.org/cgit/network/connman/connman.git/tree/doc/manager-api.txt>

Here I understand it needs an 'agent' to feed the passphrase, thus I'm afraid I cannot send it programmatically.

Do you have any recommendation about?

I guess connman has for each session a directory in /var/lib/connman/. So we have to connect manually to a new wifi, create a session and then connman connects to saved wifi automatically. But I am not sure - just guessing when observing the behavior.


Is integer vectorization accuracy / precision of integer division CPU-dependent?

I tried to vectorize the premultiplication of 64-bit colors of 16-bit integer ARGB channels.

I quickly realized that due to lack of accelerated integer division support I need to convert my values to `float` and use some SSE2/SSE4.1 intrinsics explicitly for the best performance. Still, I wanted to leave the non-specific generic version as a fallback solution (I know that it's currently slower than some vanilla operations but it would provide future compatibility for possible improvements).

However, the results are incorrect on my machine.

A very minimal repro:

    // Test color with 50% alpha
    (ushort A, ushort R, ushort G, ushort B) c = (0x8000, 0xFFFF, 0xFFFF, 0xFFFF);

    // Minimal version of the fallback logic if HW intrinsics cannot be used:
    Vector128<uint> v = Vector128.Create(c.R, c.G, c.B, 0u);
    v = v * c.A / Vector128.Create(0xFFFFu);
    var cPre = (c.A, (ushort)v[0], (ushort)v[1], (ushort)v[2]);

    // Original color:
    Console.WriteLine(c); // prints (32768, 65535, 65535, 65535)

    // Expected premultiplied color:   (32768, 32768, 32768, 32768)
    Console.WriteLine(cPre); // prints (32768, 32769, 32769, 32769)

I tried to determine what instructions are emitted causing the inaccuracy but I was really surprised to see that in SharpLab the results are [correct](https://sharplab.io/#v2:EYLgtghglgdgNAFxAJwK4wD4AEBMBGAWACgsAGAAizwDoAldBKMAU2oEkYFlYBnKAYx4BuYsQAUqHgAsA9sgTkAgnHKTZ88rRVq5CgOLbpu8gCEAlOX7kAvOTGkAHgA5SrlY4BiXj+4fef5J7eZiJExABqzPwIcng4TgA8qLAIAHzkAG425JHRsfHUAMLIzBAIzGL8dCpVBpbUJu6oIcRZtlkAVPWK5AD0OVExyHFORSVlFUFezaEZEMiWAAol2ZXUynY68mYZANqkALoqEkbbe3hHm6cIO7s4By1hJHgAnJWPVG/8y8yPQA). On the other hand, the issue is [reproducible](https://dotnetfiddle.net/4AMEZS) in .NET Fiddle.

Is it something that's expected on some platforms or should I report it in the runtime repo as a bug?

* * *

### Update

Nevermind, this is clearly a bug. Using other values cause totally wrong results:

    using System;
    using System.Numerics;
    using System.Runtime.Intrinsics;

    (ushort A, ushort R, ushort G, ushort B) c = (32768, 65535, 32768, 16384);

    Vector128<uint> v1 = Vector128.Create(c.R, c.G, c.B, 0u);
    v1 = v1 * c.A / Vector128.Create(0xFFFFu);

    // prints <32769, 49152, 57344, 0> instead of <32768, 16384, 8192, 0>
    Console.WriteLine(v1);

    // Also for the older Vector<T>
    Span<uint> span = stackalloc uint[Vector<uint>.Count];
    span[0] = c.R;
    span[1] = c.G;
    span[2] = c.B;
    Vector<uint> v2 = new Vector<uint>(span) * c.A / new Vector<uint>(0xFFFF);

    // prints <32769, 49152, 57344, 0, 0, 0, 0, 0> on my machine
    Console.WriteLine(v2);

In the end I realized that the issue was at the multiplication: if I replace `* c.A` to the constant expression `* 32768`, then the result is correct. For some reason the `ushort` value is not correctly extracted/masked(?) out from the packed field. Even `Vector.Create` is affected:

    (ushort A, ushort R, ushort G, ushort B) c = (32768, 65535, 32768, 16384);

    Console.WriteLine(Vector128.Create((int)c.A)); // -32768
    Console.WriteLine(Vector128.Create((int)32768)); // 32768
    Console.WriteLine(Vector128.Create((int)c.A, (int)c.A, (int)c.A, (int)c.A)); // 32768

* * *

### Update 2

In the end filed an [issue](https://github.com/dotnet/runtime/issues/83387) in the runtime repo

@GyörgyKőszeg: The compiler output I linked is for vec / 0xffffu.  The results are exact, using a multiplicative inverse like I said, same as compilers do for scalar uint / constant.  Why does GCC use multiplication by a strange number in implementing integer division?  Did you think I meant v >> 16?  Oh, you probably thought I meant doing the * part of you expression with an integer multiply; I was talking about using the high half of integer multiply as part of the division.

Alright, this must be a bug, using some other values end up in total off results. I will update the post soon.

@PeterCordes: yes, bit shifting is faster with vanilla code than division but the results are not exactly the same. And the floating point version now works well, it's just about the issue I noticed.

If you were using FP, it would probably perform better to multiply by 1.0f/0xffff (or however you write a float constant in C#), although that can't be represented exactly. But with about 23 bits of precision, might still give the correct 16-bit integer after rounding or truncating to 16-bit.

I don't see anything in the software fallback path which would cause this... I'd be tempted to either raise this in the dotnet/runtime repo, or ask in #allow-unsafe-blocks on discord (lots of JIT people hang out there, and they'll tell you if you need to open an issue)

This can be done with integer multiply and shifts, although it's not super efficient since it needs the high half of a 32x32 => 64-bit multiply, and SSE2 / AVX only gives you that with pmuludq which gives you the full 64-bit results.  So only half the input elements per vector.  godbolt.org/z/7E9P9aWMh shows GCC and clang using a multiplicative inverse for dividing a vector of 4x uint32_t by 0xffffu.  This is exact integer division, no FP involved at any point.

In the meantime I realized that SharpLab also produces the wrong result in Debug mode. Strange, because on my computer both Debug and Release builds are incorrect. So I start to believe this is a bug after all.


Is there something like a continuation Arrow transformer?

The [`ContT`](https://hackage.haskell.org/package/mtl-2.2.1/docs/Control-Monad-Cont.html#v:ContT) monad transformer has a interesting property: If there is a `* -> *` type such as `Set`, that has well-defined monadic operations, but can't have a `Monad` instance due to some constraints (here `Ord a`), it's possible to wrap it in `ContT` (`ContT r Set`) to get a monad instance, and defer the constraints outside it, like when we inject `Set` into `ContT r Set`. See [Constructing efficient monad instances on `Set` using the continuation monad](https://stackoverflow.com/q/12183656/1333025).

Is there something similar for arrows? An [arrow transformer](https://hackage.haskell.org/package/arrows-0.4.4.1/docs/Control-Arrow-Transformer.html#t:ArrowTransformer) that'd allow to wrap an "almost arrow" into it, getting a valid `Arrow` instance, and defer problematic constraints to the part where we inject the "almost arrow" into it?

For example, if we had a type `AlmostArrow :: * -> * -> *` for which we'd have the usual `Arrow` operations, but with constraints, such as

    arr' :: (Ord a, Ord b) => (a -> b) -> AlmostArrow a b
    (>>>') :: (Ord a, Ord b, Ord c) => AlmostArrow a b -> AlmostArrow b c -> AlmostArrow a c

As a bonus, if yes, is there some nifty, generic category-theory way how to derive both `ContT` and such an arrow transformer?

A free arrow would surely do the trick.  Then you just have to do some junk to make it efficient.