License problem: ConvertUTF is non-free, use libicu instead #349

sebastic · 2017-01-22T21:56:54Z

The lintian QA tool reported a license problem with the ConvertUTF.{c,h} files included in ncgen (license-problem-convert-utf-code):

The following file source files include material under a non-free license from Unicode Inc. Therefore, it is not possible to ship this in main or contrib.

This license does not grant any permission to modify the files (thus failing DFSG#3). Moreover, the license grant to attempt to restrict use to "products supporting the Unicode Standard" (thus failing DFSG#6).

In this case a solution is to use libicu and to remove this code by repacking.

If this is a false-positive, please report a bug against Lintian.

Refer to https://bugs.debian.org/823100 for details.

Quoting the mentioned Debian Free Software Guidelines (DFSG) paragraphs:

3. Derived Works

The license must allow modifications and derived works, and must allow them to be distributed under the same terms as the license of the original software.

6. No Discrimination Against Fields of Endeavor

The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.

Please remove the problematic ConvertUTF.{c,h} files and use libicu instead.

The text was updated successfully, but these errors were encountered:

DennisHeimbigner · 2017-01-22T22:02:34Z

Not sure I see this as a problem; AFAIK we have not modified it and since it was included
to support utf8 in netcdf-3, it meets that criteria. Is the issue transitivity? That is,
that the program using the netcdf-c library only indirectly support utf8 by using
netcdf-c? Please elaborate your concerns.
In any case, I will look at libicu.

DennisHeimbigner · 2017-01-22T22:14:36Z

Ok, so after a very quick look, the problem with libicu is that it is serious overkill
for our purposes and is way to general. We need something a very small footprint.
It appears to me that I will have to do major surgery on the source code to
extract just the parts I need. So, this switch would/will take a while; it will not happen
any time soon.

WardF · 2017-01-22T23:12:13Z

I agree libicu is overkill. On Monday I'll take a closer look at the convertutf license and see if there are other alternatives; I'll also contribute to the conversation regarding the potential problem for NetCDF that it may pose.

sebastic · 2017-01-23T07:23:42Z

The problem with the ConvertUTF code is that its license is incompatible with the license of NetCDF. The NetCDF license explicitly allows modification, which the ConvertUTF license does not.

The ghostscript bugreport linked from the Debian bugreport has more information:

According to http://unicode.org/forum/viewtopic.php?f=9&t=90 - summarized at http://stackoverflow.com/questions/2685004/why-does-unicode-org-no-longer-offer-a-reference-utf-8-16-32-converter . ConvertUTF is obsolete and buggy.

According to discussion at https://lists.debian.org/debian-legal/2006/01/msg00534.html, Richard Stallman and the Unicode consortium has noth acknowledged compatibility issues with licensing of the code - issues has been solved for later code releases issued by the Unicode consortium, but according to https://web.archive.org/web/20081228105917/http://www.unicode.org/Public/PROGRAMS/CVTUTF/ there has been no newer release of ConvertUTF since 2004.

Because NetCDF does not comply with the DFSG due to the inclusion of the ConvertUTF files which don't allow modification, NetCDF and all its reverse dependencies need to be removed from Debian & Ubuntu if this issue is not resolved. Which would be a great disservice to our users.

DennisHeimbigner · 2017-01-23T19:08:35Z

I found an alternative that claims to be the MIT license.
I have attached (below) the actual LICENSE file; Does it look acceptable?
=Dennis Heimbigner

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

sebastic · 2017-01-23T19:14:37Z

Yes, the MIT licensed alternative would be a good replacement (license-wise), since both it and the NetCDF explicitly allow modification and don't contain terms contrary to the other license.

DennisHeimbigner · 2017-02-16T03:44:18Z

I have just discovered two things. 1. At some point, the utf8proc license was modified to allow modification 2. continued development of utf8proc was taken over by the Julia Language developers. My reference is this page: https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md I will immediately shift to using this version of utf8proc. Please examine the license on the above referenced web page and let me know if it is satisfactory. =Dennis Heimbigner Unidata

…

On 1/22/2017 2:56 PM, Bas Couwenberg wrote: The lintian QA tool reported a license problem with the |ConvertUTF.{c,h}| files included in |ncgen| (license-problem-convert-utf-code <https://lintian.debian.org/tags/license-problem-convert-utf-code.html>): The following file source files include material under a non-free license from Unicode Inc. Therefore, it is not possible to ship this in main or contrib. This license does not grant any permission to modify the files (thus failing DFSG#3). Moreover, the license grant to attempt to restrict use to "products supporting the Unicode Standard" (thus failing DFSG#6). In this case a solution is to use libicu and to remove this code by repacking. If this is a false-positive, please report a bug against Lintian. Refer to https://bugs.debian.org/823100 for details. Quoting the mentioned Debian Free Software Guidelines (DFSG) paragraphs: *3. Derived Works* The license must allow modifications and derived works, and must allow them to be distributed under the same terms as the license of the original software. *6. No Discrimination Against Fields of Endeavor* The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research. Please remove the problematic |ConvertUTF.{c,h}| files and use |libicu| instead. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#349>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA3P23pcx1GbHm8BdAglMkLAqB-pb4gbks5rU9CngaJpZM4LqhMu>.

sebastic · 2017-02-16T06:51:46Z

Unfortunately the Unicode data license is non-free due to the advertising clause (like BSD-4-Clause).

DennisHeimbigner · 2017-02-16T16:31:21Z

Interesting. You are aware, I presume that libicu also has this same restriction. Hence
we cannot use that either. In fact, my guess is that all utf software suffers from this same
problem.

sebastic · 2017-02-16T17:35:29Z

I was not aware of that icu used the same license terms, since the icu license terms were apparently deemed acceptable for Debian main by the FTP masters (although that's no precedent), it's probably fine to adopt the utf8proc from Julia. If they reject the netcdf upload due to those license terms I'll raise that issue then.

It turns out that the utf8proc software we are using was turned over to the Julia Language developers and the license terms changed to allow modification. (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). So the fix here is as follows: 1. Wrap the library with a fixed interface: libdispatch/dutf8.c and include/ncutf8.h. 2. Replace the existing utf8proc code with the new version from https://github.com/JuliaLang/utf8proc. 3. Add a couple more test cases: nc_test/tst_utf8_validate.c and nc_test_utf8_phrases.c. If/when I can find a usable normalization test, I will incorporate that later.

DennisHeimbigner · 2017-02-16T18:20:31Z

ok

…

On 2/16/2017 10:35 AM, Bas Couwenberg wrote: I was not aware of that icu used the same license terms, since the icu license terms were apparently deemed acceptable for Debian main by the FTP masters (although that's no precedent), it's probably fine to adopt the utf8proc from Julia. If they reject the netcdf upload due to those license terms I'll raise that issue then. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#349 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA3P26pTAUzJFEdeoVi8vN8VEbfd7FKBks5rdIjhgaJpZM4LqhMu>.

Update utf8proc.[ch] to use the version now maintained by the Julia Language project (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). The license for the previous version was unacceptable for the Debian and Ubuntu release systems. The new version both updates the code and addresses the license issue. It turns out that the utf8proc software we are using was turned over to the Julia Language developers and the license terms changed to allow modification. (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). So the fix here is as follows: 1. Wrap the library with a fixed interface: libdispatch/dutf8.c and include/ncutf8.h. 2. Replace the existing utf8proc code with the new version from https://github.com/JuliaLang/utf8proc. 3. Add a couple more test cases: nc_test/tst_utf8_validate.c and nc_test_utf8_phrases.c. If/when I can find a usable normalization test, I will incorporate that later.

WardF · 2017-03-24T21:57:46Z

This issue is resolved, closing.

sebastic · 2017-06-06T05:33:48Z

ncgen/ConvertUTF.c & ncgen/ConvertUTF.h are still included in 4.5.0-rc1, please re-open this issue and remove/replace those files.

WardF · 2017-06-06T16:56:24Z

@DennisHeimbigner Can the solution you provided for libdispatch/ in #364 also be applied in ncgen/?

DennisHeimbigner · 2017-06-06T18:23:39Z

I did not remember that this code was being used in ncgen. I will take responsibility for it.
Also, odd because it means we are still including the old code?

WardF · 2017-06-06T18:56:15Z

The old code (convertUTF.c/h) is currently only in ncgen; it was removed from libdispatch and the new code was put in place. I looked at libicu and I'm glad you found this solution as libicu is not practical for our purposes; it is too large, too difficult to deploy, and is an unnecessary dependency.

If you have yet to create a branch to work from, would you ~~~fork~~~ branch from v4.5.0-release-branch? If it's too late, no worries, I will make the necessary merges.

DennisHeimbigner · 2017-06-06T19:00:13Z

Ok, I will fork the release branch. This is going to be harder than I thought.
The old convert code was used only to convert utf8 to utf16 for java. The new
code apparently has no utf16 support. Since I sincerely doubt that the cdl->java
code is being used, I may take the easy way out.

WardF · 2017-06-06T19:02:21Z

Ok, I will fork the release branch. This is going to be harder than I thought.
The old convert code was used only to convert utf8 to utf16 for java. The new
code apparently has no utf16 support. Since I sincerely doubt that the cdl->java
code is being used, I may take the easy way out.

To make sure I understand, it was only used to convert utf8 to utf16 when having ncgen generate Java code? If this is the case I'd be loathe to rip it out completely as that is very useful, maybe just leave the hooks in and commented out or something. I dug into this a bit and it wouldn't be impossible to write our own converter if need be. But having this functionality removed for the next release candidate wouldn't be a problem. And would give people a chance to speak up if they need/rely on this.

WardF · 2017-06-06T19:03:49Z

Also, thanks for forking that branch; I've set it up so that anything in that branch can propagate downstream into a release candidate as well as upstream back into master, but the inverse would be messy.

DennisHeimbigner · 2017-06-06T19:52:04Z

It turns out that I do have utf8 -> utf32 conversion. And converting
utf32 -> utf16 can be approximated by truncating the 32bits to 16 bits.
I will put in an error for when the approximation fails. In any case, this
fix should be "good enough".

WardF · 2017-07-14T16:45:19Z

@DennisHeimbigner Is this issue ready to be closed out? I think it is but I thought I'd double check before closing it.

WardF · 2017-07-14T16:45:46Z

Actually, the fix was merged so closing this out, I'll reopen if I hear I need to.

WardF added this to the 4.4.2 milestone Feb 16, 2017

WardF assigned DennisHeimbigner and WardF Feb 16, 2017

DennisHeimbigner mentioned this issue Feb 16, 2017

Resolve license issue with the utf8proc code. #364

Merged

WardF closed this as completed Mar 24, 2017

WardF reopened this Jun 6, 2017

WardF closed this as completed Jul 14, 2017

lighterowl mentioned this issue May 1, 2020

replace libtransmission/ConvertUTF.{c,h} with supported alternative transmission/transmission#612

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

License problem: ConvertUTF is non-free, use libicu instead #349

License problem: ConvertUTF is non-free, use libicu instead #349

sebastic commented Jan 22, 2017 •

edited

Loading

DennisHeimbigner commented Jan 22, 2017

DennisHeimbigner commented Jan 22, 2017

WardF commented Jan 22, 2017

sebastic commented Jan 23, 2017

DennisHeimbigner commented Jan 23, 2017

sebastic commented Jan 23, 2017

DennisHeimbigner commented Feb 16, 2017 via email

sebastic commented Feb 16, 2017

DennisHeimbigner commented Feb 16, 2017

sebastic commented Feb 16, 2017

DennisHeimbigner commented Feb 16, 2017 via email

WardF commented Mar 24, 2017

sebastic commented Jun 6, 2017

WardF commented Jun 6, 2017

DennisHeimbigner commented Jun 6, 2017

WardF commented Jun 6, 2017 •

edited

Loading

DennisHeimbigner commented Jun 6, 2017

WardF commented Jun 6, 2017

WardF commented Jun 6, 2017

DennisHeimbigner commented Jun 6, 2017

WardF commented Jul 14, 2017

WardF commented Jul 14, 2017

License problem: ConvertUTF is non-free, use libicu instead #349

License problem: ConvertUTF is non-free, use libicu instead #349

Comments

sebastic commented Jan 22, 2017 • edited Loading

DennisHeimbigner commented Jan 22, 2017

DennisHeimbigner commented Jan 22, 2017

WardF commented Jan 22, 2017

sebastic commented Jan 23, 2017

DennisHeimbigner commented Jan 23, 2017

I found an alternative that claims to be the MIT license. I have attached (below) the actual LICENSE file; Does it look acceptable? =Dennis Heimbigner

sebastic commented Jan 23, 2017

DennisHeimbigner commented Feb 16, 2017 via email

sebastic commented Feb 16, 2017

DennisHeimbigner commented Feb 16, 2017

sebastic commented Feb 16, 2017

DennisHeimbigner commented Feb 16, 2017 via email

WardF commented Mar 24, 2017

sebastic commented Jun 6, 2017

WardF commented Jun 6, 2017

DennisHeimbigner commented Jun 6, 2017

WardF commented Jun 6, 2017 • edited Loading

DennisHeimbigner commented Jun 6, 2017

WardF commented Jun 6, 2017

WardF commented Jun 6, 2017

DennisHeimbigner commented Jun 6, 2017

WardF commented Jul 14, 2017

WardF commented Jul 14, 2017

sebastic commented Jan 22, 2017 •

edited

Loading

I found an alternative that claims to be the MIT license.
I have attached (below) the actual LICENSE file; Does it look acceptable?
=Dennis Heimbigner

WardF commented Jun 6, 2017 •

edited

Loading