Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

colordiff fails on many input colorant types #556

Closed
adrhill opened this issue Sep 25, 2024 · 1 comment
Closed

colordiff fails on many input colorant types #556

adrhill opened this issue Sep 25, 2024 · 1 comment

Comments

@adrhill
Copy link

adrhill commented Sep 25, 2024

Working on DitherPunk.jl, colordiff errors on many input types.
It is unclear to me whether these errors are intended or whether we could simply convert inputs to the colorspace the metric uses. If this behavior is unintended, I can address it in a PR.

MWE

Running fuzzy tests on a subset of Colorants reveals that no metric supports XYZ. Testing on more "obscure" colorant types reveals even more errors.

METRICS = (DE_2000, DE_94, DE_JPC79, DE_CMC, DE_BFD, DE_AB, DE_DIN99, DE_DIN99d, DE_DIN99o)
COLORTYPES = (RGB, HSV, XYZ, Lab, Gray)

@testset "Fuzzy testing" begin
    @testset "$M" for M in METRICS
        metric = M()
        @testset "$(A), $(B)" for (A, B) in Iterators.product(COLORTYPES, COLORTYPES)
            # Sample random inputs
            for _ = 1:100
                a = rand(A)
                b = rand(B)
                @test_nowarn colordiff(a, b; metric=metric)
            end
        end
    end
end

which returns

Test Summary:  |   Pass  Error   Total   Time
Fuzzy testing  | 144000     81  144081  16.9s
  DE_2000      |  16000      9   16009   1.9s
    RGB, RGB   |   1000           1000   0.1s
    HSV, RGB   |   1000           1000   0.1s
    XYZ, RGB   |             1       1   0.0s
    Lab, RGB   |   1000           1000   0.1s
    Gray, RGB  |   1000           1000   0.1s
    RGB, HSV   |   1000           1000   0.1s
    HSV, HSV   |   1000           1000   0.1s
    XYZ, HSV   |             1       1   0.0s
    Lab, HSV   |   1000           1000   0.1s
    Gray, HSV  |   1000           1000   0.1s
    RGB, XYZ   |             1       1   0.0s
    HSV, XYZ   |             1       1   0.0s
    XYZ, XYZ   |             1       1   0.0s
    Lab, XYZ   |             1       1   0.0s
    Gray, XYZ  |             1       1   0.0s
    RGB, Lab   |   1000           1000   0.1s
    HSV, Lab   |   1000           1000   0.1s
    XYZ, Lab   |             1       1   0.0s
    Lab, Lab   |   1000           1000   0.1s
    Gray, Lab  |   1000           1000   0.1s
    RGB, Gray  |   1000           1000   0.1s
    HSV, Gray  |   1000           1000   0.1s
    XYZ, Gray  |             1       1   0.0s
    Lab, Gray  |   1000           1000   0.1s
    Gray, Gray |   1000           1000   0.1s
  DE_94        |  16000      9   16009   1.8s
    RGB, RGB   |   1000           1000   0.1s
    HSV, RGB   |   1000           1000   0.1s
    XYZ, RGB   |             1       1   0.0s
    Lab, RGB   |   1000           1000   0.1s
    Gray, RGB  |   1000           1000   0.1s
    RGB, HSV   |   1000           1000   0.1s
    HSV, HSV   |   1000           1000   0.1s
    XYZ, HSV   |             1       1   0.0s
    Lab, HSV   |   1000           1000   0.1s
    Gray, HSV  |   1000           1000   0.1s
    RGB, XYZ   |             1       1   0.0s
    HSV, XYZ   |             1       1   0.0s
    XYZ, XYZ   |             1       1   0.0s
    Lab, XYZ   |             1       1   0.0s
    Gray, XYZ  |             1       1   0.0s
    RGB, Lab   |   1000           1000   0.1s
    HSV, Lab   |   1000           1000   0.1s
    XYZ, Lab   |             1       1   0.0s
    Lab, Lab   |   1000           1000   0.1s
    Gray, Lab  |   1000           1000   0.1s
    RGB, Gray  |   1000           1000   0.1s
    HSV, Gray  |   1000           1000   0.1s
    XYZ, Gray  |             1       1   0.0s
    Lab, Gray  |   1000           1000   0.1s
    Gray, Gray |   1000           1000   0.1s
  DE_JPC79     |  16000      9   16009   1.9s
    RGB, RGB   |   1000           1000   0.1s
    HSV, RGB   |   1000           1000   0.1s
    XYZ, RGB   |             1       1   0.0s
    Lab, RGB   |   1000           1000   0.1s
    Gray, RGB  |   1000           1000   0.1s
    RGB, HSV   |   1000           1000   0.1s
    HSV, HSV   |   1000           1000   0.1s
    XYZ, HSV   |             1       1   0.0s
    Lab, HSV   |   1000           1000   0.1s
    Gray, HSV  |   1000           1000   0.1s
    RGB, XYZ   |             1       1   0.0s
    HSV, XYZ   |             1       1   0.0s
    XYZ, XYZ   |             1       1   0.0s
    Lab, XYZ   |             1       1   0.0s
    Gray, XYZ  |             1       1   0.0s
    RGB, Lab   |   1000           1000   0.1s
    HSV, Lab   |   1000           1000   0.1s
    XYZ, Lab   |             1       1   0.0s
    Lab, Lab   |   1000           1000   0.1s
    Gray, Lab  |   1000           1000   0.1s
    RGB, Gray  |   1000           1000   0.1s
    HSV, Gray  |   1000           1000   0.1s
    XYZ, Gray  |             1       1   0.0s
    Lab, Gray  |   1000           1000   0.1s
    Gray, Gray |   1000           1000   0.1s
  DE_CMC       |  16000      9   16009   1.8s
    RGB, RGB   |   1000           1000   0.1s
    HSV, RGB   |   1000           1000   0.1s
    XYZ, RGB   |             1       1   0.0s
    Lab, RGB   |   1000           1000   0.1s
    Gray, RGB  |   1000           1000   0.1s
    RGB, HSV   |   1000           1000   0.1s
    HSV, HSV   |   1000           1000   0.1s
    XYZ, HSV   |             1       1   0.0s
    Lab, HSV   |   1000           1000   0.1s
    Gray, HSV  |   1000           1000   0.1s
    RGB, XYZ   |             1       1   0.0s
    HSV, XYZ   |             1       1   0.0s
    XYZ, XYZ   |             1       1   0.0s
    Lab, XYZ   |             1       1   0.0s
    Gray, XYZ  |             1       1   0.0s
    RGB, Lab   |   1000           1000   0.1s
    HSV, Lab   |   1000           1000   0.1s
    XYZ, Lab   |             1       1   0.0s
    Lab, Lab   |   1000           1000   0.1s
    Gray, Lab  |   1000           1000   0.1s
    RGB, Gray  |   1000           1000   0.1s
    HSV, Gray  |   1000           1000   0.1s
    XYZ, Gray  |             1       1   0.0s
    Lab, Gray  |   1000           1000   0.1s
    Gray, Gray |   1000           1000   0.1s
  DE_BFD       |  16000      9   16009   1.9s
    RGB, RGB   |   1000           1000   0.1s
    HSV, RGB   |   1000           1000   0.1s
    XYZ, RGB   |             1       1   0.0s
    Lab, RGB   |   1000           1000   0.1s
    Gray, RGB  |   1000           1000   0.1s
    RGB, HSV   |   1000           1000   0.1s
    HSV, HSV   |   1000           1000   0.1s
    XYZ, HSV   |             1       1   0.0s
    Lab, HSV   |   1000           1000   0.1s
    Gray, HSV  |   1000           1000   0.1s
    RGB, XYZ   |             1       1   0.0s
    HSV, XYZ   |             1       1   0.0s
    XYZ, XYZ   |             1       1   0.0s
    Lab, XYZ   |             1       1   0.0s
    Gray, XYZ  |             1       1   0.0s
    RGB, Lab   |   1000           1000   0.1s
    HSV, Lab   |   1000           1000   0.1s
    XYZ, Lab   |             1       1   0.0s
    Lab, Lab   |   1000           1000   0.1s
    Gray, Lab  |   1000           1000   0.1s
    RGB, Gray  |   1000           1000   0.1s
    HSV, Gray  |   1000           1000   0.1s
    XYZ, Gray  |             1       1   0.0s
    Lab, Gray  |   1000           1000   0.1s
    Gray, Gray |   1000           1000   0.1s
  DE_AB        |  16000      9   16009   1.9s
    RGB, RGB   |   1000           1000   0.1s
    HSV, RGB   |   1000           1000   0.1s
    XYZ, RGB   |             1       1   0.0s
    Lab, RGB   |   1000           1000   0.1s
    Gray, RGB  |   1000           1000   0.1s
    RGB, HSV   |   1000           1000   0.1s
    HSV, HSV   |   1000           1000   0.1s
    XYZ, HSV   |             1       1   0.0s
    Lab, HSV   |   1000           1000   0.1s
    Gray, HSV  |   1000           1000   0.1s
    RGB, XYZ   |             1       1   0.0s
    HSV, XYZ   |             1       1   0.0s
    XYZ, XYZ   |             1       1   0.0s
    Lab, XYZ   |             1       1   0.0s
    Gray, XYZ  |             1       1   0.0s
    RGB, Lab   |   1000           1000   0.1s
    HSV, Lab   |   1000           1000   0.1s
    XYZ, Lab   |             1       1   0.0s
    Lab, Lab   |   1000           1000   0.1s
    Gray, Lab  |   1000           1000   0.1s
    RGB, Gray  |   1000           1000   0.1s
    HSV, Gray  |   1000           1000   0.1s
    XYZ, Gray  |             1       1   0.0s
    Lab, Gray  |   1000           1000   0.1s
    Gray, Gray |   1000           1000   0.1s
  DE_DIN99     |  16000      9   16009   1.9s
    RGB, RGB   |   1000           1000   0.1s
    HSV, RGB   |   1000           1000   0.1s
    XYZ, RGB   |             1       1   0.0s
    Lab, RGB   |   1000           1000   0.1s
    Gray, RGB  |   1000           1000   0.1s
    RGB, HSV   |   1000           1000   0.1s
    HSV, HSV   |   1000           1000   0.1s
    XYZ, HSV   |             1       1   0.0s
    Lab, HSV   |   1000           1000   0.1s
    Gray, HSV  |   1000           1000   0.1s
    RGB, XYZ   |             1       1   0.0s
    HSV, XYZ   |             1       1   0.0s
    XYZ, XYZ   |             1       1   0.0s
    Lab, XYZ   |             1       1   0.0s
    Gray, XYZ  |             1       1   0.0s
    RGB, Lab   |   1000           1000   0.1s
    HSV, Lab   |   1000           1000   0.1s
    XYZ, Lab   |             1       1   0.0s
    Lab, Lab   |   1000           1000   0.1s
    Gray, Lab  |   1000           1000   0.1s
    RGB, Gray  |   1000           1000   0.1s
    HSV, Gray  |   1000           1000   0.1s
    XYZ, Gray  |             1       1   0.0s
    Lab, Gray  |   1000           1000   0.1s
    Gray, Gray |   1000           1000   0.1s
  DE_DIN99d    |  16000      9   16009   1.9s
    RGB, RGB   |   1000           1000   0.1s
    HSV, RGB   |   1000           1000   0.1s
    XYZ, RGB   |             1       1   0.0s
    Lab, RGB   |   1000           1000   0.1s
    Gray, RGB  |   1000           1000   0.1s
    RGB, HSV   |   1000           1000   0.1s
    HSV, HSV   |   1000           1000   0.1s
    XYZ, HSV   |             1       1   0.0s
    Lab, HSV   |   1000           1000   0.1s
    Gray, HSV  |   1000           1000   0.1s
    RGB, XYZ   |             1       1   0.0s
    HSV, XYZ   |             1       1   0.0s
    XYZ, XYZ   |             1       1   0.0s
    Lab, XYZ   |             1       1   0.0s
    Gray, XYZ  |             1       1   0.0s
    RGB, Lab   |   1000           1000   0.1s
    HSV, Lab   |   1000           1000   0.1s
    XYZ, Lab   |             1       1   0.0s
    Lab, Lab   |   1000           1000   0.1s
    Gray, Lab  |   1000           1000   0.1s
    RGB, Gray  |   1000           1000   0.1s
    HSV, Gray  |   1000           1000   0.1s
    XYZ, Gray  |             1       1   0.0s
    Lab, Gray  |   1000           1000   0.1s
    Gray, Gray |   1000           1000   0.1s
  DE_DIN99o    |  16000      9   16009   1.9s
    RGB, RGB   |   1000           1000   0.1s
    HSV, RGB   |   1000           1000   0.1s
    XYZ, RGB   |             1       1   0.0s
    Lab, RGB   |   1000           1000   0.1s
    Gray, RGB  |   1000           1000   0.1s
    RGB, HSV   |   1000           1000   0.1s
    HSV, HSV   |   1000           1000   0.1s
    XYZ, HSV   |             1       1   0.0s
    Lab, HSV   |   1000           1000   0.1s
    Gray, HSV  |   1000           1000   0.1s
    RGB, XYZ   |             1       1   0.0s
    HSV, XYZ   |             1       1   0.0s
    XYZ, XYZ   |             1       1   0.0s
    Lab, XYZ   |             1       1   0.0s
    Gray, XYZ  |             1       1   0.0s
    RGB, Lab   |   1000           1000   0.1s
    HSV, Lab   |   1000           1000   0.1s
    XYZ, Lab   |             1       1   0.0s
    Lab, Lab   |   1000           1000   0.1s
    Gray, Lab  |   1000           1000   0.1s
    RGB, Gray  |   1000           1000   0.1s
    HSV, Gray  |   1000           1000   0.1s
    XYZ, Gray  |             1       1   0.0s
    Lab, Gray  |   1000           1000   0.1s
    Gray, Gray |   1000           1000   0.1s
ERROR: Some tests did not pass: 144000 passed, 0 failed, 81 errored, 0 broken.
@adrhill adrhill mentioned this issue Sep 25, 2024
Draft
@adrhill
Copy link
Author

adrhill commented Sep 25, 2024

Sorry, it looks like this actually failed because rand isn't defined on all colorants.

@adrhill adrhill closed this as completed Sep 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@adrhill