CHANGES

Pike 8.1: Changes since Pike 8.0 (scratch area for future release notes)
------------------------------------------------------------------------

New language features
---------------------

o Unlimited character constant size.

  There is no longer any limit to the size of character constants,
  e.g. 'acdefghijk' creates the bignum 0x61636465666768696a6b.

o 'auto' type added.

  This is much like a strict typed mixed. The actual type is deduced
  compile-time.

  Typical use is as the type for variables in foreach when looping over
  complexly typed values, or as the type in macro-defined functions and
  similar.

  auto can be used as return type for a function, it will be the
  type-union of all the types of the return statements in the
  function.

o typeof(X) can now be used as a type.

  The main use case is in macros, but it might be useful for other
  things as well (as an example in a typedef).

  typedef typeof(Val.true)|typeof(Val.false) bool;

  typeof, when used like this, behaves very much like the C++
  decltype() expression.

o ** operator added. It's exponentiation and works with most
  combination of numerical types (int,float,Gmp.mpq,Gmp.mpf,Gmp.mpz)

  `** and ``** operator overloading functions added.

  This introduces one incompatible change: Previously the "pow"
  function called a "_pow" function in the first argument if it was an
  object. It has now been changed to also use `** (or, rather, pow()
  is now implemented using predef::`**()).

o Three pass compiler.

  An extra pass of the compiler has been added. It is used to properly
  resolve types in forward referring expressions, and is only run when
  needed.

o #pragma disassemble

  This directive is now available even in pikes compiled --without-debug.

  The byte code output is now interleaved with the generated machine code
  on selected architectures.

o Complain about redundant backslash escapes.

o '__weak__' modifier added.

  It is now possible to declare object variables to have weak references.

o Support for arguments to implicit lambdas added.

  Implicit lambdas now behave as if they were declared as
  lambda(mixed ... __ARGS__).

  Support for this feature can be detected with #ifdef __ARGS__.

o Function local function declarations are now statements
  (and not expressions). This means that there is no longer
  any need to terminate them with a semicolon.

o Anonymous class definitions are now always expressions.

o Function local class definitions are now statements
  (and not expressions). This means that there is no longer
  any need to terminate them with a semicolon.

o Complain about shadowed variant functions.

o Machine code support for more architectures.

  There's now machine code support for arm32 and arm64.

o Fixed multiple integer over- and underflow bugs.

o Extended sscanf %O

  sscanf() is now able to parse all base types that sprintf %O can
  output. It can now also parse most literal constants pike accepts,
  including integers prefixed with 0x and 0b, and floating point
  numbers in scientific notation.

o Returning void

  It is now possible to return void responses from void functions.

    void a() { return; }
    void b() { return a(); }

o Protocols.DNS now supports encoding and decoding CAA RRs.


Bug fixes
---------

o Operator functions

  Calling operator functions with more than two arguments will now
  work correctly for objects, where previously only the first two
  objects where added.

  When adding arrays, multisets and mappings, UNDEFINED will now
  always be ignored. Previously it was only ignored when first in the
  argument list, otherwise an exception would be thrown.

  The LFUNs will now be called with a consistent number of
  arguments. Pike implementations rarely implemented varargs on
  operator LFUNs, so this change should address many potential hidden
  errors.

o Protocols.HTTP.Server

  The server module will now read payloads for HTTP PUT requests, just
  as any other method. Previously it would stop reading the body and
  it was up to the caller to read enough data from the socket and
  combine with already read data.

  Setting "connection" header in the "extra_heads" to Request object
  method response_and_finish() will now control if keep-alive should be
  used or not. Otherwise it will be decided by the clients request
  headers, as previously.

  The headers "content-type", "content-range", "content-length",
  "server", "date" and "last-modified" will not be added or
  overwritten if provided in the "extra_heads".

  Header names in "extra_heads" will not have their case modified.


Incompatible changes
--------------------

o Sql.Sql is no longer a wrapper class, but a function.

  The wrapper class has been obsoleted by introduction of the
  new base class __builtin.Sql.Connection. Note that it is
  still possible to use Sql.Sql as the type of connection objects.

o Gz.crc32 and Nettle.crc32c now only return positive results.

o glob() has changed.

  The main incompatibilities are that [ and \ are now special
  characters in the pattern, and if you check the return value against
  1 instead of checking if it's true, and use an array as the first
  argument you will have to change your code.

  This is in order to extend the glob function to cover what
  'standard' glob functions do:

  glob() now accepts quotes (\* to match a single *, as an example)
  and handles ranges ([abc] for a, b or c, [a-z0-9] for a single
  character between a and z or 0-9

  You can also negate a range using ^ or ! ([^a-zA-Z] as an example).

  When the first argument (the pattern) to glob is an array, glob now
  returns which pattern in the array matched.

o hash() has been changed to use siphash. The old hash function is
  available under the name hash_8_0(). Note that this is not a
  cryptographic hash function.

o Stdio.UDP()->send() no longer throws errors on EMSGSIZE and EWOULDBLOCK.

o If a thread exits with by throwing, the thrown value is propagated to wait().

o String.trim_all_whites() renamed String.trim().

o Floats are no longer by default sorted before added. This may reduce
  the precision but increases the speed of adding large number of
  floats by a factor of 500. Applications handling floats with large
  differences in magnitude need to apply the appropriate sorting
  before arithmetics. As `+ was the only operator that performed
  sorting, and functions like Array.sum did not, this was already a
  concern.

o Returning UNDEFINED from `+ and ``+ is not allowed and will cause an
  exception.

o RegGetValue(), RegGetKeyNames(), RegGetKeyValues(), openlog(),
  syslog() and closelog() have been moved to System.


New modules
-----------

o ADT.Scheduler & ADT.TreeScheduler

  These are variants of ADT.Heap where elements typically aren't
  removed from the heap.

o Apple.Keychain

  Parser for Apple Keychain format files (like the ones in
  /System/Library/Keychains/).

o __builtin.Sql

  Generic base classes for Sql API modules.

o Filesystem.Zip
o Function.bind()

  Partially evaluate a function. Similar to Function.curry, but with
  better granularity.

o Standards.HPack (RFC 7541)
o Stdio.FakePipe

  A simulated Stdio.Pipe.

o Parser.Markdown

o Crypto.Checksum

  This module collect non-cryptographic checksums. Support for crc32,
  adler32 and Castagnoli CRC (CRC32C).

  NB: In the future these may be amended to support the Crypto.Hash API.

o Parser.ECMAScript

  This module simply provides a token splitter for
  ECMAScript/JavaScript.

o Web.EngineIO & Web.SocketIO
o Protocols.HTTP2
o Bittorrent.DHT
o Standards.MsgPack
o Web.Auth & Web.Api


New features
------------

o predef::equal()

  equal() on functions now checks if their definitions are same identifier
  in the same program.

o predef::gc()

  gc() called with a weak mapping as argument now removes weak references
  that are only held by that mapping.

o ADT.Heap

  - An indirection object ADT.Heap.Element has been added to make it
    possible to optimize several heap operations.

  - Added low_pop().

o Crypto & Nettle

  - Added Curve25519 and EdDSA25519.

o Filesystem.Monitor

  The filesystem monitoring system now uses accelleration via
  Inotify et al.

o Gmp

  - mpf is now implemented using gmpf if the library is available.

  - Improved support for GMP 5.0 and later.

o GTK2

  Multiple runtime fixes.

o JOSE (JSON Object Signing and Encryption)

  Some low-level API support has been added to the Crypto and Web
  modules to support parts of RFC 7515 - 7520.

o MasterObject

  - Protect against the same file being compiled concurrently
    in multiple threads.

  - cast_to_program() and cast_to_object() should now be thread safe.

o CompilerEnvironment()->lock()

  Access to the compiler lock.

o Crypto.ECC.Curve.Point

  A point on an ECC curve.

o Parser.Pike

  Support new language features.

o Protocols.DNS.server

  Derived classes can now override report_decode_error() and
  handle_decode_error() to change how errors while decoding a DNS
  packet are reported and handled.

o Protocols.WebSocket

  Multiple API changes.

o Random rewrite

  The random functions have been rewritten to ensure security by
  default. random_string() and random() now get their data directly
  from the operating system random generator, i.e. /dev/urandom on
  most unixen. This is about half the speed compared with the
  random_string function in Pike 8.0, but is now as secure as the
  system random generator.

  For consumers of random data that have additional requirements,
  different random generators are exposed in the new module
  Random. The module has the following generators:

  - Random.Interface

    This is not actually a generator, but an interface class that is
    inherited into all the other generators. It contains code that can
    turn the output from the random_string method into random numbers
    with different limits without introducing bias. It also contains
    code for all the different variants of random() on different
    types. This is currently not possible to implement in Pike code,
    as the typing is too complicated and it is not possible to access
    private/protected _random methods in objects.

  - Random.System

    This generator maps directly on top of the system random
    generator. This is the default generator used for random() and
    random_string().

  - Random.Deterministic

    This generator creates the same sequence of random numbers for a
    given seed, with good pseudo random properties.

  - Random.Hardware

    This generator accesses the hardware random generator, when
    available.

  - Random.Fast

    This generator takes entropy from the Random.System, but feeds
    it into a cryptographic pseudo random number generator to be
    able to output data fast. This is not the default random number
    generator to avoid loading crypto code on every startup.

  Comparing the different generators with each other gives the
  following approximate speeds on a Linux system with hardware
  random support:

    Random.System           1.0
    Pike 8.0 random_string  0.45
    Random.Hardware         0.25
    Random.Fast             0.20
    Random.Deterministic    0.20

  Objects implementing the _random lfun now get two arguments, the
  current random_string() and random() functions. This is convenient
  for C-level functions that doesn't have to look up functions
  themselves. Note that it is possible for a user to replace these
  with non-conforming functions (returning values of the wrong type,
  strings of the wrong length or shift size, and values outside the
  given range) or even non-functions.

  All code in Pike that uses random now uses the current random
  functions (though in some cases fixed at object creation). This
  allows for repeatable results if the random functions are replaced
  with a deterministic random generator, such as
  Random.Deterministic. Example:

    Random.Deterministic rnd = Random.Deterministic( seed );
    add_constant( "random_string", rnd->random_string );
    add_constant( "random", rnd->random );

o Sql

  - Most Sql C-modules converted to cmod.

  - Added next_result().

  - ODBC & tds: Support more datatypes.

  - ODBC: Support big_typed_query().

  - pgsql: Lots of changes and fixes.

o SSL

  - Support session tickets.

  - Support Negotiated FF-DHE.

  - Support client certificates.

  - Support ALPN.

  - Prefer AEAD suites to CBC suites.

  - SSL.File supports set_buffer_mode().

o Standards.PKCS

  Support PKCS#8 private keys.

o String.Buffer & Stdio.Buffer

  Added _search().

o The self testing framework now supports *.test-files.

o Thread

  - _sprintf() improvements: Thread.Mutex now prints the ID of the thread
    holding the lock, and thread IDs are shown as hexadecimal numbers.

  - Thread.Farm now supports a callback for thread creation and termination
    for the purpose of tracking thread names.

o sprintf %x

  %X and %x can now be used on 8-bit wide strings to get a hexadecimal
  representation of their contents. Just calling sprintf("%x",data) is
  the same as calling String.string2hex(data).

o Unicode 10.0.0.

o Unicode.is_whitespace()

  This new function returns if a unicode character is a whitespace
  characer or not.


Deprecated symbols and modules
------------------------------

o Sql.mysql_result and Sql.mysqls_result have been deprecated.
  Use Sql.Result instead.

o call_function() has been deprecated. Use `()() instead.


Removed features and modules
----------------------------

o Compatibility for Pike versions before 7.8 is no longer available.

o GTK1 library is deprecated, so glue code is removed.


C-level API changes
-------------------

o The contract for functions is now changed so that a function is no
  longer required to clean the stack. The topmost value of the stack
  will be regarded as the return value and the rest of the items on
  the stack, compared to before the function arguments were pushed,
  will be popped and discarded. Efuns still have to clean their stack
  as previously.

o Removed add_function, pike_add_function, pike_add_function2,
  simple_add_variable, map_variable and MAP_VARIABLE. This removes the
  remaining few API:s where text types were used. Use ADD_FUNCTION and
  PIKE_MAP_VARIABLE instead.

o Removed the functions my_rand and my_srand. Use the random functions
  on the stack for _random lfuns, the push_random_string or look up
  the random function from get_builtin_constants(). For deterministic
  pseudo random, create a private Random.Deterministic object.

o The preprocessor has been converted into a cmod, and been modified
  to use more standard Pike datatypes.

o The preprocessor-specific hashtable implementation has been removed.

o The gdb_backtraces() function is now available also --without-debug.

o There's now support to block mapping hashtables from being shrunk
  on map_delete().

o guaranteed_memset() is replaced with secure_zero() which fills a
  buffer with zero. On x86 SSE2 is used to zero the memory without
  loading it into the CPU cache, as this function is typically used
  before calling free() on memory with cryptographic key data.


Documentation
-------------

o RFC references added.

o Character encoding issues fixed.

o Added @enum/@endenum markup.

o Support undocumented enums with documented constants.


Building and installing
-----------------------

o GMP 4.1 or later is now required.

o C99 assumed

  The configure tests will not check for functions defined in C99
  anymore and C99 compiler support is assumed.

Optimizations
-------------

o Removed the GC marker hash table. For types which require GC markers,
  they are now allocated as parf of the data type. This significantly
  improves GC performance ( up to a factor if 2 in some situations ).