get/set all values referred by an optic

[aplavin]

Some existing optics, such as Elements(), If() and Recursive(), can only be modifyed - one cannot retrieve the values specified by them. I understand that it's not obvious how to implement optic(obj) for them and even what type it should return. However, from time to time I try to reach for this functionality: either to specifically extract the corresponding values, or just to see what values modify would be called with.

Do you think it's possible to design value extraction for those optics in a consistent manner?

[jw3126]

A quick and dirty way to do this is:

out = Any[]
modify(obj, optic) do val
    push!(out, val)
    f(val)
end

@rafaqz has done a lot of work in this direction, see #23
Ultimately we could not find a performant way to implement this, that's why the PR is not merged.

[aplavin]

#23 looks a bit over my head - not even talking about implementation, I don't fully understand the interface and usecases.
Extracting values with Elements and If is probably much easier to implement performantly than very general #23 queries. A major issue is the interface: it should be consistent with

Accessors.jl/src/testing.jl

Lines 18 to 24 in fa0a446

	function test_modify_law(f, lens, obj)
	obj_modify = modify(f, obj, lens)
	old_val = lens(obj)
	val = f(old_val)
	obj_setfget = set(obj, lens, val)
	@test obj_modify == obj_setfget
	end

[jw3126]

Okay, so is your question whether one can implement optic(obj) for these such that the lens laws are satisfied?

I think that is not really possible. For instance:

julia> using Accessors

julia> obj = [1,2,3]
3-element Vector{Int64}:
 1
 2
 3

julia> optic = Elements()
Elements()

julia> modify(identity, obj, optic)
3-element Vector{Int64}:
 1
 2
 3

julia> set(obj, optic, 1)
3-element Vector{Int64}:
 1
 1
 1

That means you can only ever construct constant vectors using set, Elements(), but you can produce arbitrary vectors using modify, Elements(). So there is no hope to decompose modify into set and "get".

Mathematically there are many flavors of optics and these can have different APIs and laws. Lens is one with lots of structure and laws. Elements() only satisfies the functor laws and does not have a "get" operation:

• @assert modify(identity, optic, obj) == obj
• @assert modify(f ∘g, optic, obj) == modify(f, optic, modify(g, optic, obj))

[rafaqz]

#23 looks a bit over my head - not even talking about implementation, I don't fully understand the interface and usecases.
Extracting values with Elements and If is probably much easier to implement performantly than very general #23 queries. A major issue is the interface: it should be consistent with

Accessors.jl/src/testing.jl

Lines 18 to 24 in fa0a446

	function test_modify_law(f, lens, obj)
	obj_modify = modify(f, obj, lens)
	old_val = lens(obj)
	val = f(old_val)
	obj_setfget = set(obj, lens, val)
	@test obj_modify == obj_setfget
	end

The use case is reconstructing arbitrary immutable objects with new/modified field values.

https://github.com/rafaqz/ModelParameters.jl is the best example of how powerful this can be.

You can also use it to e.g. replace Arrays deeply nested in wrapper types with GPUArrays. You dont need to know or specify the fields, just the types to replace.

You can set values using this method, by passing in a tuple. But the order of the matched field types in the object tree determines the order objects are replaced in.

[aplavin]

The use case is reconstructing arbitrary immutable objects with new/modified field values.

That describes the whole Accessors.jl :)

With simple lenses in Accessors, it's pretty clear when to use them and what's the benefit. More complex optics, such as Elements(), require some getting used to, but they remain relatively transparent. I personally never used Recursive(), which is probably the most advanced optic here - just never encountered a reasonable usecase myself.

You can also use it to e.g. replace Arrays deeply nested in wrapper types with GPUArrays. You dont need to know or specify the fields, just the types to replace.

That sounds like a job for the Recursive() optic, right? Descend wherever possible and process what you see.
What would #23 bring on top of it?

[aplavin]

@jw3126 thanks for the detailed explanation!

Do you think there's space for functions like get_all(obj, optic) and set_all(obj, optic, values) that extract all specified values as some sort of collection? Other than Recursive, this shouldn't be too hard to implement efficiently, I think.

Optics have great composability, so that one learns about a few concepts and they (often) seamlessly combine together. Hoping for something similar with "multivalued" optics, and a function like get_all...

[jw3126]

If you manage to implement a fast get_all that would be super awesome. In #23 we were not able to implement a get_all that type inference liked.

[rafaqz]

That describes the whole Accessors.jl :)

Maybe you miss the part about arbitrary objects. Most of Accessors works on objects where you know a field name or some other detail. But sometimes we don't know anything about the structure of the objects or the names of any of fields, and want to make generic changes to it.

Maybe its already possible with accessors? But Im not sure how.

If you can define a function that takes the Float64 from any bits type object of any nesting depth and makes them all Float32 in the reconstructed object or a flat Tuple - and any other similar transformation - calculated at compile time. That would remove the need for that PR.

How youre describing get_all and set_all is essentially what my PR does, and it composes with other optics.

Edit: I may not have been clear but being able to get/set objects from a Tuple is a key part of that PR, rather than just replacing things as you walk an object. Some other package like Optim.jl may be in the middle of the transformation wanting a vector of floats, so Recursive is not going to be enough.

[aplavin]

Yes, I understand the usecase "replace all values of this type to this type" now: like the GPU conversion or Float bitness.
Probably, I even wanted something like this, once or twice. Getting (or setting?) all values specified by an optic is a more common case, regularly wish this existed.

How youre describing get_all and set_all is essentially what my PR does, and it composes with other optics.

Looked once again into that PR, still not sure if I fully understand the interface. What do you mean "composes with other optics"? When would one use Query with another optic?

I would say a more intuitive approach wouldn't even involve introducing a new type: just define getall/setall on existing optics, including Recursive(). Of course, this entails the same type stability issues (hopefully will resolve in a later Julia?), but here I'm only talking about the interface.

[rafaqz]

I dont realy get what is different with your proposal to that PR or what you dont understand at this point 😆

It has to be able to replace Flatten.jl and power ModelParameters.jl, as I linked earlier. That was my main goal, with some nice @setall syntax on top.

By composes I mean you could do something like @setall of the val fields of every United.Quantity in an object, using an optic. I cant remember the syntax to do that, but there are examples in the thread and the tests for the PR.

Maybe you should try it out rather than reinvent that wheel, if you will likely have the same type stability issue.

[aplavin]

Suppose I have an optic like opt = @optic _.a |> Elements() |> _.b |> Properties(). Does Query provide a way to get all elements referred to by this optic, as in getall(obj, opt)? That, and the corresponding setall, is basically everything I'd need.

I haven't tried implementing getall yet, do you think the type stability issues remain even without arbitrary recursion? I mean limiting getall to simple lenses and optics like Elements or Properties - for now.

[rafaqz]

I don't know how Elements works really, but you can do things like this with arbitrary optics where _[2].g is:

@getall (x[2].g for x in missings_obj if x isa NamedTuple)
@setall (x[2].g for x in missings_obj if x isa NamedTuple) = 5

This example gets all theg properties of the second value in every NamedTuple in any object , or sets them all to equal 5.

You can do this manually without the macros using Query and passing it your optic.

See:
https://github.com/rafaqz/Accessors.jl/blob/queries/test/test_queries.jl

[aplavin]

A major point of composability is the ability to just pass an optic, and let the inner function do whatever it wants with the values specified by that optic. Currently, the values can always be modified, sometimes can be get and set. My proposal is to extend this interface with getall and setall -- for all optics!

Query seems to solve a somewhat different, arguably more specialized, problem. I don't really want to recurse indefinitely and find all NamedTuples!
And again, I believe it doesn't require adding yet another optic -- the current set seems rich enough to cover the example usecases.

[rafaqz]

"Get me every Float64 in this thing" seems like incredibly generic functionality to me ;)

[aplavin]

Sorry, do you agree with something, or arguing with something from my previous message?