-
Notifications
You must be signed in to change notification settings - Fork 76
Nothing cols fixes #1551
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Nothing cols fixes #1551
Conversation
…performance of allNulls() in some cases
…urate than `Unit` as getting something out of this column will always result in an exception, not "just run", like `Unit` does. Added emptyOf for if you need a specific type
…dded redundancy checks for nothing types in column creation
|
|
||
| public class UpdateException(override val message: String, cause: Throwable? = null) : | ||
| IllegalStateException(message, cause), | ||
| DataFrameError |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Better to add this supertype only if you have example of code that triggers this in compiler plugin
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, that might be difficult, as it most often relies on runtime information. (besides maybe update {}.with { error() }). I thought we now preferred to use the DataFrameError type for any new exception we create
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think about it as a way to opt-in reporting warnings for exceptions, that we know for sure make sense for user. In theory we could report any exception. DataFrameError will help us to validate them first
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
soo... DataFrameWarning? ;P
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could be
| * Creates an empty [DataColumn] with given [name]. | ||
| * @see emptyOf | ||
| */ | ||
| public fun empty(name: String = ""): DataColumn<Nothing> = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have two operations that use empty: concatImpl and getColumn. Please see if they can be affected by this change. Otherwise it's quite neat
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seems concat fails on dataframes without rows, but not because of Nothing columns, but because it uses ::class reflection in createColumnGuessingType(), interesting
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed it in the next commit :)
This PR contains a handful of fixes and optimizations regarding
Nothing(?)in column types:Nothing?column is detected asFormattedFrame<*>?in notebooks #1546 and adds a helpful note tocolsOfto prevent future confusionDataColumn.empty()created aUnit-typed column. I have no idea why, I made itNothing, which better represents thatemptyCol.get(0): Nothingwill always throw an exception. AddedDataColumn.emptyOf<T>()if you need a specifically typed empty column.updateprovided little info, so I addedUpdateExceptionfor this.::class-logic totypeOf's (for efficiency), and added redundancy checks:Nothingcolumn, it can only do that if it has no values; so the resulting column must be empty)Nothing?column, it can only do that if all its values arenull)I also checked usages of
colsOf()querying for nullable types to see if similar issues like #1546 existed. Fortunately, in most cases, including null-filled columns are not an issue! All statistics functions simply skip nulls, for instance.There are some unfortunate cases, like:
dataFrameOf( "a" to columnOf(1, 2, 3, null), "b" to columnOf(null, null, null, null), "c" to columnOf(7, 3, 2, 65), ).fillNulls { colsOf<Int?>() }.with { 0 }which throw
java.lang.IllegalArgumentException: Can not add value of class kotlin.Int to column of type kotlin.Nothing?. Value = 0because columnb: Nothing?is included bycolsOf<Int?>(). However, hopefully, the note to add.filter { !it.allNulls() }is found :). Alternatively, we could decide to add an argumentcolsOf<Int?>(exactType = true)in the future.