Array notation design considerations

This article details the design considerations for array notation in APL. Feedback may be given via the Discussion page or any other array language chat room or forum, for example under the Array Notation Discussion topic on the Dyalog Forums. This design considerations page, and if necessary the specification, will be kept up to date as feedback is received.

Objectives

The following requirements were proposed as objectives for an APL array notation:

1. No new glyphs
2. Reusing existing glyphs for similar purposes
3. Similarity to other languages (K, JSON, CSS)
4. Visual attractiveness
5. Intuitive syntax
6. As little syntactic sugar as possible

Specific considerations

Various alternatives have been considered and the following details each design decision.

Glyphs

The design requirement for no new glyphs was contentious, and both bi-glyph and non-ASCII brackets were considered. Bi-glyphs were rejected out of readability concerns, especially when nested. For example, 1 1 3⍴2 could have been written as [[[[2 2 2]]]]. Non-ASCII brackets were rejected for font and keyboarding reasons, as well as to make it easier for non-APL systems to generate APL data. For example, ⟦…⟧ was proposed to denote a collection of major cells, forming a new array of rank one-higher than the rank of the highest-rank constituent cell. However, few fonts support these glyphs.

The eventual choice was to go with existing symbols, and this had important implications for the specifics of the notation. While ideally, a notation would have been introduced for a collection of major cells, thereby handling both vectors and higher-rank arrays, a problem presents itself with axes of length 1, because both square brackets and round parentheses already have meaning with when surrounding a single statement (namely function axis/bracket indexing and precedence/function trains). Thus, while 2 ⟦3⟧ could have denoted the nested array 2 (1⍴3), this isn't viable with 2 [1⍴3] because this already denotes indexing 2 using the indices 1⍴3. To disambiguate, at least one statement separator or line break must be present in each level of array notation brackets and parentheses.

Disambiguating square brackets

The overloading of square brackets, currently in use only for function axis and bracket indexing, to mean a higher-rank array, poses a problem of disambiguation in the case where there is only one major cell. For example 'abc'[3 3] could be equivalent to 'cc' or 'abc'(1 2⍴3) depending on whether the brackets are interpreted as indexing or an array. Two proposals have been made, and it is possible to support either or both:

1. Square brackets are interpreted as representing an array if no other interpretation is possible, e.g. immediately following an opening round parenthesis, curly brace, or square bracket, or beginning a statement.
2. Square brackets are interpreted as representing an array if they are "broken", i.e. contain a diamond or newline that isn't enclosed in another round parenthesis, curly brace, or square bracket.

Option 1 depends on an outer context of the notation, while option 2 depends on the inner content of the notation. The latter has similarity to the manner in which a dfn is determined to be a function, a monadic operator, or a dyadic operator: If the curly braces contain ⍵⍵ then the dfn is a dyadic operator; otherwise, a ⍺⍺ indicates a monadic operator; and any other dfn is a function.

Minimum rank of major cells

While ⟦⟦3⟧⟧ could denote 1 1⍴3 using non-ASCII glyphs, an equivalent ASCII scheme instead would have required [[3⋄]⋄] where the inner bracket creates a vector, and the outer creates a matrix. Using line breaks instead of diamonds, it was found to be counter-intuitive that

[
 3
 5
  ]

was to denote two-element vector while

[
 3 4
 5 6
    ]

would be a two-row matrix. This is indeed the case in dzaima/APL, as opposed to in Dyalog APL, where a special rule was added to the effect that in such collections of major cells, every cell will be considered to have a rank of at least 1, even if it is a scalar. However, this choice introduced the need for a separate notation to allow vectors to be written over multiple lines, and therefore the round parentheses were extended from their traditional use in strand notation to also denote collections of enclosed elements.

Name-value pairs

As a notation for namespaces, several details were debated, as detailed below.

Separators between name-value pairs

Should ⋄ or ; be used to separate name-value pairs (in addition to line breaks)?

The ⋄ was chosen to separate name-value pairs, as it is generally exchangeable with a line break, while ; though it is used to separate names ― without values ― in headers and in locals lines. Furthermore, it was seen as natural the values would be computed in reading order (left-to-right) just like multiple statements are, and while ⋄ would imply this, ; wouldn't. Indeed, in the statement A[B;C], expression C is evaluated before expression B. It was briefly considered to have values computed from the right, just line stranding is, but this was rejected because replacing the semi-colons with line breaks would then require evaluation beginning with the last line and working upwards!

Namespace delimiters

Should round parentheses ((…)) or square brackets ([…]) be used to enclose namespaces?

Round parentheses were chosen because namespaces are seen as (unordered) lists, and so are more similar to vectors than higher-rank arrays. Furthermore, [] already had meaning (indexing all elements of a vector) while () didn't have any existing use, and so could be used to denote a new empty namespace, equivalent to ⎕NS 0⍴⊂''.

Separator between name and value

Should : or ← separate the name from the value?

While initially, ← was seen as the obvious choice to separate the name and the value, it was soon discovered that a namespace with only one member would be indistinguishable from a parenthesised assignment. Furthermore, it was noted that value expressions could contain intermediary assignments, and that such assignments were of a fundamentally different nature from the name-value declaration. The intermediary assignments would happen in a temporary scope, with any created variables disappearing once the namespace member value was established.

Scoping

In which scope the value expressions should be evaluated?

Value expressions could be evaluated in the newly established namespace (similar to expressions in :Namespace scripts), or in the surrounding scope (similar to inline expressions in JavaScript's object notation). It was envisioned that a main usage of the literal notation would be to collect existing values into a namespace, and evaluating inside the new namespace would force the use of ##. to fetch values in the surrounding scope. In a departure from JavaScript, it was found most natural that such intermediate assignments be local to the value expression, similar to assignments in dfns. Global assignment is still available using ⎕THIS.name←value, just as in dfns.

After publication of the formal proposal, Peter Mikkelsen, a then recently hired Dyalog Ltd employee, pointed out an inconsistency between value expressions in namespace and vector notation:

g←1 ⋄ (a:g←2 ⋄ b:g) would make (a:2 ⋄ b:1) but g remains 1

while

g←1 ⋄ (g←2 ⋄ g) would make (2 ⋄ 2) updating g to be 2.

Because of this, the specification was changed on May 5, 2023 to follow JavaScript; that assignments inside value expressions then affect the surrounding scope. Auto-localisation of temporary variables used in building the value can then be achieved by wrapping the expression in an anonymous dfn.

Timeline

1996

The publication of John Scholes' Dynamic Functions in Dyalog APL showed that a number of expressions could be grouped together within paired delimiters and separated by line-ends. This hinted at the possibility of doing a similar thing between brackets and parentheses to solve a problem for which Phil Last had been seeking a solution for a decade.

2010

At the 2010 APL Conference in Berlin Dyalog introduced experimental interpreter APL#. This included a notation for namespaces as name-value pairs between paired double brackets with major and minor separators being line-end and assignment arrow: [[name←value ⋄ ...]]; and a sort of extended, multiple expression separated by line-ends and between paired parentheses.

2013

Phil Last sent a proposal to Dyalog outlining two possible executable notations for creating multi-dimensional arrays without function application. One using potential new system construct :Array and :Cell to be used in tradfns and another using line-ends between balanced brackets to define arrays of rank-2 or greater in both dfns and tradfns.

It became RFE 9458: Large and higher rank literal values. See File:Embedding data.pdf

Description: Proposal for a mechanism to specify large and higher rank literal values directly in code.

After which in the following year ...

2014

At Dyalog '14, Morten Kromberg said:

The emphasis on using scripts to store source code means that it's probably time for us to come up with a notation for constants in the language so that in your script you can declare matrices and so on in a nice readable fashion.

Although no concrete proposal was made at the time, he set the expectation of this being the subject of a presentation the following year.

2015

At Dyalog '15, Phil Last explained that he considered the lack of such a notation a big hole in APL notation and gave a suggestions for such a notation. He presented a model using square brackets to indicate collections of major cells of rank 1 or higher, delimited by line breaks and/or diamonds, for example [1 2 3 ⋄ 4 5 6] would be equivalent to 2 3⍴1 2 3 4 5 6. He also proposed that if the delimited expressions were assignments, then the notation would instead declare members of an anonymous namespace, for example for example [a←3 ⋄ b←6]. He pointed out that this overloading of the symbols meant that the array notation could only represent constants, as allowing general expressions would lead to ambiguity. He also mentioned that doubled symbols or Unicode brackets could be used instead.

After the presentation, Phil Last had a conversation with Adám Brudzewsky who had recently joined Dyalog Ltd., the language developer of Dyalog APL, and who was inspired to begin an internal Dyalog research project on the matter. Meanwhile, Acre Desktop, a project manager that Last co-develops, moved from storing APL items in component files to storing them in text files, necessitating a literal notation for arrays, and his notation for arrays was adopted. Acre stores unscripted namespaces as directories, so the need for a literal namespace notation only arises when a namespace is an element in a larger array, something that is quite unlikely for application constants.

2016

Phil Last published a more formal proposal in the Vector Journal where he emphasised the need for such a notation to become an integral part of the language. He added escape sequences to strings, further distancing the notation from compatibility with existing APL code.

Array notation at Dyalog '17.

2017

At Dyalog '17, Adám Brudzewsky proposed an alternative notation using round parentheses to indicate collections of major cells of any rank, thus allowing the notation to express nested vectors though scalar major cells, for example (⊂1 2 3 ⋄ ⊂4 5 6) would be equivalent to (1 2 3)(4 5 6). This notation had a striking similarity to the informal notation used in the NARS reference manual over 35 years prior. For namespace, he proposed using colon (:) to delimit name-value pairs, inspired by JSON in which colon is used in the same manner, despite assignment being denoted by = in JavaScript, from which JSON was derived. This distinction allowed arbitrary expressions in arrays, opening the possibility of full integration into the language, while also allowing a namespace with no members to be denoted (). Last's proposal required [:] to distinguish it from bracket indexing into a vector while eliding the indices, a technique used to address all elements.

In addition to the main array notation, Brudzewsky also proposed allowing line breaks between quotes in strings to represent a vector of character vectors (with leading and trailing spaces trimmed). While not included in the live presentation, Brudzewsky's slide deck included a discussion of whether expressions resulting in a scalar should be treated as singleton vectors or not. It concluded that if they were treated as vectors, then an alternative notation in the form of a line continuation character would be necessary to allow writing large vectors over multiple lines of code.

Array notation at Dyalog '18.

2018

At Dyalog '18, Adám Brudzewsky returned with a solution to the issue on whether scalars should be regarded as 1-element vectors (thus increasing the rank of the containing array) or left as scalars (thus forming a vector). He reintroduced square brackets as collections of major cells of rank 1 or higher, repurposing round parentheses as vectors.

The namespace notation remained as before, using round parentheses so the empty namespace could be written in a consistent manner, but he presented formalised scoping rules for the value expressions, namely that these would run in the surrounding namespace, but within their own scope, so any assignment done during such an expression. For example (a:b,b←1 2) would neither populate the new namespace with a member b, nor create such a variable in the global scope. Acre quickly adopted this notation.

Array notation at Dyalog '20

2020

In the spring of 2020, dzaima/APL adopted the proposed array notation with the exception of forcing the result of statements in square brackets to rank 1 or higher.

At Dyalog '20, Adám Brudzewsky presented the notation as Release Candidate 1 and showed how Dyalog APL 18.0's updated version of Link (a simple interface for using source code in text files, synchronising the file system and the workspace) includes experimental support the array notation, including a facility to use multi-line array notation inside functions. He estimated that Dyalog APL 20.0 would include native interpreter support for the notation in 2022.

APL Germany's 2020 journal also included a description of the notation, including a discussion of potential issues with assignment.

2021

At Dyalog '21, Morten Kromberg implied that comunity feedback would have been solicited, and implementation would begin in time for Dyalog APL 19.0. However, this proved a bit optimistic.

Array notation at Dyalog '22

2022

At Dyalog '22, Morten Kromberg announced that Dyalog Ltd was close to having the notation working.

2023

On April 21, 2023, Dyalog Ltd published a blog post by Morten Kromberg announcing to the community the formal proposal for an APL array notation.

On May 5, 2023, the specification for scoping in namespaces was changed due to feedback from Dyalog Ltd employee Peter Mikkelsen. Assignments inside value expressions would now affect the surrounding scope rather than having dfn-like auto-localisation, which can instead be achieved by wrapping the expression in an anonymous dfn.

Comparison of array notations

The following systems support list or vector notation in some form, beyond simple strand notation. The separators ; in A+ and K, and ⋄ in APL and BQN, indicate any separator, including a line break.

System	Vectors	High-rank	Namespaces	Function arrays	Assignable
Nial	[,]	No	N/A	Special	No
A+	(;)	No	N/A	First-class	Yes
K	(;)	N/A	[key:val;]	First-class	Yes
BQN	⟨⋄⟩	[⋄]	{key⇐val⋄}	First-class	Yes
dzaima/APL	(⋄)	[⋄]	(key:val⋄)	Special	No
Dyalog Link	(⋄)	[⋄]	(key:val⋄)	No	No
Acre Desktop	(⋄)	[⋄]	[key←val⋄]	No	N/A
TinyAPL	⟨⋄⟩	[⋄]	⦃key←val⋄⦄	First-class	Yes

Nial and A+ do not support namespaces, while K does not support high-rank arrays, so any such notation is not applicable. The "Function arrays" column indicates whether functions can be placed in array notation. "First class" indicates that functions are first class, so this is possible without special consideration. "Special" indicates creating a special vectors of functions that can be applied to arguments to return a list of results. The "Assignable" column indicates that array notation can be used as an assignment target to perform destructuring. BQN's namespaces don't use a dedicated construction; instead, any block (like a dfn) with ⇐ statements returns a namespace reference. Acre Desktop only uses array notation for storing literal arrays; it cannot appear in executable code.