Browse Source

Merge remote-tracking branch 'upstream/master' into hardened/current/master

hardened/current/master
Shawn Webb 9 months ago
parent
commit
e1f5f1c597
Signed by untrusted user: shawn.webb <shawn.webb@hardenedbsd.org> GPG Key ID: FF2E67A277F8E1FA
100 changed files with 2804 additions and 32148 deletions
  1. +2
    -3
      Makefile.inc1
  2. +1
    -19
      Makefile.libcompat
  3. +3
    -0
      RELNOTES
  4. +7
    -0
      UPDATING
  5. +21
    -7
      cddl/contrib/opensolaris/cmd/zpool/zpool.8
  6. +45
    -0
      cddl/contrib/opensolaris/cmd/zpool/zpool_main.c
  7. +2
    -0
      cddl/contrib/opensolaris/lib/libzfs/common/libzfs.h
  8. +21
    -0
      cddl/contrib/opensolaris/lib/libzfs/common/libzfs_pool.c
  9. +23
    -11
      cddl/contrib/opensolaris/lib/libzfs_core/common/libzfs_core.c
  10. +3
    -0
      cddl/contrib/opensolaris/lib/libzfs_core/common/libzfs_core.h
  11. +36
    -0
      contrib/libc++/include/cmath
  12. +24
    -21
      contrib/libc++/include/random
  13. +12
    -11
      contrib/libxo/Makefile.am
  14. +3
    -0
      contrib/libxo/README.md
  15. +3
    -1
      contrib/libxo/configure.ac
  16. +14
    -60
      contrib/libxo/doc/Makefile.am
  17. +98
    -98
      contrib/libxo/doc/api.rst
  18. +11
    -3
      contrib/libxo/doc/conf.py
  19. +269
    -0
      contrib/libxo/doc/encoders.rst
  20. +3
    -4
      contrib/libxo/doc/faq.rst
  21. +71
    -71
      contrib/libxo/doc/field-formatting.rst
  22. +20
    -20
      contrib/libxo/doc/field-modifiers.rst
  23. +46
    -44
      contrib/libxo/doc/field-roles.rst
  24. +1
    -0
      contrib/libxo/doc/index.rst
  25. +0
    -27436
      contrib/libxo/doc/libxo-manual.html
  26. +0
    -3990
      contrib/libxo/doc/libxo.txt
  27. +51
    -51
      contrib/libxo/doc/options.rst
  28. +9
    -0
      contrib/libxo/doc/top-link.html.in
  29. +32
    -9
      contrib/libxo/doc/xo.rst
  30. +7
    -7
      contrib/libxo/doc/xohtml.rst
  31. +443
    -39
      contrib/libxo/doc/xolint.rst
  32. +7
    -7
      contrib/libxo/doc/xopo.rst
  33. +5
    -2
      contrib/libxo/encoder/Makefile.am
  34. +3
    -2
      contrib/libxo/encoder/cbor/enc_cbor.c
  35. +51
    -0
      contrib/libxo/encoder/csv/Makefile.am
  36. +826
    -0
      contrib/libxo/encoder/csv/enc_csv.c
  37. +1
    -1
      contrib/libxo/encoder/test/enc_test.c
  38. +1
    -1
      contrib/libxo/libxo/add.man.in
  39. +63
    -99
      contrib/libxo/libxo/libxo.c
  40. +2
    -2
      contrib/libxo/libxo/xo_buf.h
  41. +30
    -5
      contrib/libxo/libxo/xo_encoder.c
  42. +54
    -2
      contrib/libxo/libxo/xo_encoder.h
  43. +17
    -1
      contrib/libxo/tests/core/Makefile.am
  44. +50
    -1
      contrib/libxo/tests/core/saved/test_01.E.out
  45. +0
    -0
      contrib/libxo/tests/core/saved/test_01.Ecsv1.err
  46. +17
    -0
      contrib/libxo/tests/core/saved/test_01.Ecsv1.out
  47. +0
    -0
      contrib/libxo/tests/core/saved/test_01.Ecsv2.err
  48. +10
    -0
      contrib/libxo/tests/core/saved/test_01.Ecsv2.out
  49. +0
    -0
      contrib/libxo/tests/core/saved/test_01.Ecsv3.err
  50. +17
    -0
      contrib/libxo/tests/core/saved/test_01.Ecsv3.out
  51. +1
    -1
      contrib/libxo/tests/core/saved/test_01.H.out
  52. +52
    -0
      contrib/libxo/tests/core/saved/test_01.HIPx.out
  53. +52
    -0
      contrib/libxo/tests/core/saved/test_01.HP.out
  54. +1
    -1
      contrib/libxo/tests/core/saved/test_01.J.out
  55. +42
    -0
      contrib/libxo/tests/core/saved/test_01.JP.out
  56. +8
    -0
      contrib/libxo/tests/core/saved/test_01.T.out
  57. +1
    -1
      contrib/libxo/tests/core/saved/test_01.X.out
  58. +40
    -0
      contrib/libxo/tests/core/saved/test_01.XP.out
  59. +1
    -1
      contrib/libxo/tests/core/saved/test_02.E.out
  60. +1
    -1
      contrib/libxo/tests/core/saved/test_03.E.out
  61. +1
    -1
      contrib/libxo/tests/core/saved/test_04.E.out
  62. +1
    -1
      contrib/libxo/tests/core/saved/test_05.E.out
  63. +1
    -1
      contrib/libxo/tests/core/saved/test_06.E.out
  64. +1
    -1
      contrib/libxo/tests/core/saved/test_07.E.out
  65. +1
    -1
      contrib/libxo/tests/core/saved/test_08.E.out
  66. +1
    -1
      contrib/libxo/tests/core/saved/test_09.E.out
  67. +1
    -1
      contrib/libxo/tests/core/saved/test_10.E.out
  68. +1
    -1
      contrib/libxo/tests/core/saved/test_11.E.out
  69. +4
    -4
      contrib/libxo/tests/core/saved/test_12.E.err
  70. +1
    -1
      contrib/libxo/tests/core/saved/test_12.E.out
  71. +4
    -4
      contrib/libxo/tests/core/saved/test_12.H.err
  72. +4
    -4
      contrib/libxo/tests/core/saved/test_12.HIPx.err
  73. +4
    -4
      contrib/libxo/tests/core/saved/test_12.HP.err
  74. +4
    -4
      contrib/libxo/tests/core/saved/test_12.J.err
  75. +4
    -4
      contrib/libxo/tests/core/saved/test_12.JP.err
  76. +4
    -4
      contrib/libxo/tests/core/saved/test_12.T.err
  77. +4
    -4
      contrib/libxo/tests/core/saved/test_12.X.err
  78. +4
    -4
      contrib/libxo/tests/core/saved/test_12.XP.err
  79. +50
    -8
      contrib/libxo/tests/core/test_01.c
  80. +8
    -7
      contrib/libxo/tests/core/test_02.c
  81. +3
    -2
      contrib/libxo/tests/core/test_03.c
  82. +9
    -8
      contrib/libxo/tests/core/test_08.c
  83. +9
    -8
      contrib/libxo/tests/core/test_09.c
  84. +9
    -8
      contrib/libxo/tests/core/test_10.c
  85. +4
    -3
      contrib/libxo/tests/core/test_11.c
  86. +11
    -10
      contrib/libxo/tests/core/test_12.c
  87. +4
    -3
      contrib/libxo/tests/gettext/gt_01.c
  88. +1
    -1
      contrib/libxo/tests/gettext/saved/gt_01.H.out
  89. +2
    -2
      contrib/libxo/tests/gettext/saved/gt_01.HIPx.out
  90. +2
    -2
      contrib/libxo/tests/gettext/saved/gt_01.HP.out
  91. +1
    -1
      contrib/libxo/tests/gettext/saved/gt_01.J.out
  92. +2
    -2
      contrib/libxo/tests/gettext/saved/gt_01.JP.out
  93. +2
    -2
      contrib/libxo/tests/gettext/saved/gt_01.T.out
  94. +1
    -1
      contrib/libxo/tests/gettext/saved/gt_01.X.out
  95. +2
    -2
      contrib/libxo/tests/gettext/saved/gt_01.XP.out
  96. +1
    -0
      contrib/libxo/tests/xo/saved/xo_02.H.err
  97. +1
    -0
      contrib/libxo/tests/xo/saved/xo_02.HIPx.err
  98. +1
    -0
      contrib/libxo/tests/xo/saved/xo_02.HP.err
  99. +1
    -0
      contrib/libxo/tests/xo/saved/xo_02.J.err
  100. +1
    -0
      contrib/libxo/tests/xo/saved/xo_02.JP.err

+ 2
- 3
Makefile.inc1 View File

@@ -803,11 +803,10 @@ XCFLAGS+= --sysroot=${WORLDTMP}
XCFLAGS+= ${BFLAGS}
.endif

.if ${MK_LIB32} != "no" && (${TARGET_ARCH} == "amd64" || \
${TARGET_ARCH} == "powerpc64" || ${TARGET_ARCH:Mmips64*} != "")
.if ${MK_LIB32} == "yes"
_LIBCOMPAT= 32
.include "Makefile.libcompat"
.elif ${MK_LIBSOFT} != "no" && ${TARGET_ARCH:Marmv[67]*} != ""
.elif ${MK_LIBSOFT} == "yes"
_LIBCOMPAT= SOFT
.include "Makefile.libcompat"
.endif


+ 1
- 19
Makefile.libcompat View File

@@ -118,28 +118,10 @@ build${libcompat}: .PHONY
.endfor
${_+_}cd ${.CURDIR}; \
${LIBCOMPATWMAKE} -f Makefile.inc1 -DNO_FSCHG libraries
.if ${libcompat} == "32"
.for _t in ${_obj} all
.if !defined(NO_RTLD)
${_+_}cd ${.CURDIR}/libexec/rtld-elf; PROG=ld-elf32.so.1 ${LIBCOMPATWMAKE} \
-DNO_FSCHG DIRPRFX=libexec/rtld-elf/ ${_t}
.endif
${_+_}cd ${.CURDIR}/usr.bin/ldd; PROG=ldd32 ${LIBCOMPATWMAKE} \
DIRPRFX=usr.bin/ldd ${_t}
.endfor
.endif

distribute${libcompat} install${libcompat}: .PHONY
.for _dir in ${_LC_LIBDIRS.yes}
${_+_}cd ${.CURDIR}/${_dir}; ${LIBCOMPATIMAKE} ${.TARGET:S/${libcompat}$//}
.endfor
.if ${libcompat} == "32"
.if !defined(NO_RTLD)
${_+_}cd ${.CURDIR}/libexec/rtld-elf; \
PROG=ld-elf32.so.1 ${LIBCOMPATIMAKE} ${.TARGET:S/32$//}
.endif
${_+_}cd ${.CURDIR}/usr.bin/ldd; PROG=ldd32 ${LIBCOMPATIMAKE} \
${.TARGET:S/32$//}
.endif

.endif
.endif # !targets(__<${_this:T}>__)

+ 3
- 0
RELNOTES View File

@@ -10,6 +10,9 @@ newline. Entries should be separated by a newline.

Changes to this file should not be MFCed.

r354517:
iwm(4) now supports most Intel 9260, 9460 and 9560 Wi-Fi devices.

r354269:
sqlite3 is updated to sqlite3-3.30.1.



+ 7
- 0
UPDATING View File

@@ -26,6 +26,13 @@ NOTE TO PEOPLE WHO THINK THAT FreeBSD 13.x IS SLOW:
disable the most expensive debugging functionality run
"ln -s 'abort:false,junk:false' /etc/malloc.conf".)

20191107:
The nctgpio and wbwd drivers have been moved to the superio bus.
If you have one of these drivers in a kernel configuration, then
you should add device superio to it. If you use one of these drivers
as a module and you compile a custom set of modules, then you should
add superio to the set.

20191021:
KPIs for network drivers to access interface addresses have changed.
Users need to recompile NIC driver modules together with kernel.


+ 21
- 7
cddl/contrib/opensolaris/cmd/zpool/zpool.8 View File

@@ -27,7 +27,7 @@
.\"
.\" $FreeBSD$
.\"
.Dd February 20, 2019
.Dd November 7, 2019
.Dt ZPOOL 8
.Os
.Sh NAME
@@ -193,6 +193,9 @@
.Ar ...
.Op Ar interval Op Ar count
.Nm
.Cm sync
.Oo Ar pool Oc Ns ...
.Nm
.Cm upgrade
.Op Fl v
.Nm
@@ -599,8 +602,8 @@ Each pool has several properties associated with it. Some properties are
read-only statistics while others are configurable and change the behavior of
the pool. The following are read-only properties:
.Bl -tag -width "dedupratio"
.It Sy alloc
Amount of storage space within the pool that has been physically allocated.
.It Sy allocated
Amount of storage space used within the pool.
.It Sy capacity
Percentage of pool space used. This property can also be referred to by its
shortened column name, "cap".
@@ -650,8 +653,6 @@ Information about unsupported features that are enabled on the pool.
See
.Xr zpool-features 7
for details.
.It Sy used
Amount of storage space used within the pool.
.El
.Pp
The space usage properties report actual physical space available to the
@@ -1573,8 +1574,8 @@ Comma-separated list of properties to display. See the
section for a list of valid properties. The default list is
.Sy name ,
.Sy size ,
.Sy used ,
.Sy available ,
.Sy allocated ,
.Sy free ,
.Sy checkpoint ,
.Sy expandsize ,
.Sy fragmentation ,
@@ -1915,6 +1916,19 @@ block size or disabled features will not be included.
.El
.It Xo
.Nm
.Cm sync
.Oo Ar pool Oc Ns ...
.Xc
Forces all in-core dirty data to be written to the primary pool storage and
not the ZIL.
It will also update administrative information including quota reporting.
Without arguments,
.Nm zpool Cm sync
will sync all pools on the system.
Otherwise, it will only sync the specified
.Ar pool .
.It Xo
.Nm
.Cm upgrade
.Op Fl v
.Xc


+ 45
- 0
cddl/contrib/opensolaris/cmd/zpool/zpool_main.c View File

@@ -100,6 +100,8 @@ static int zpool_do_history(int, char **);
static int zpool_do_get(int, char **);
static int zpool_do_set(int, char **);

static int zpool_do_sync(int, char **);

/*
* These libumem hooks provide a reasonable set of defaults for the allocator's
* debugging facilities.
@@ -144,6 +146,7 @@ typedef enum {
HELP_GET,
HELP_SET,
HELP_SPLIT,
HELP_SYNC,
HELP_REGUID,
HELP_REOPEN
} zpool_help_t;
@@ -200,6 +203,7 @@ static zpool_command_t command_table[] = {
{ "history", zpool_do_history, HELP_HISTORY },
{ "get", zpool_do_get, HELP_GET },
{ "set", zpool_do_set, HELP_SET },
{ "sync", zpool_do_sync, HELP_SYNC },
};

#define NCOMMAND (sizeof (command_table) / sizeof (command_table[0]))
@@ -285,6 +289,8 @@ get_usage(zpool_help_t idx)
"[<device> ...]\n"));
case HELP_REGUID:
return (gettext("\treguid <pool>\n"));
case HELP_SYNC:
return (gettext("\tsync [pool] ...\n"));
}

abort();
@@ -2648,6 +2654,45 @@ error:
return (err ? 1 : 0);
}

/*
* zpool sync [-f] [pool] ...
*
* -f (undocumented) force uberblock (and config including zpool cache file)
* update.
*
* Sync the specified pool(s).
* Without arguments "zpool sync" will sync all pools.
* This command initiates TXG sync(s) and will return after the TXG(s) commit.
*
*/
static int
zpool_do_sync(int argc, char **argv)
{
int ret;
boolean_t force = B_FALSE;

/* check options */
while ((ret = getopt(argc, argv, "f")) != -1) {
switch (ret) {
case 'f':
force = B_TRUE;
break;
case '?':
(void) fprintf(stderr, gettext("invalid option '%c'\n"),
optopt);
usage(B_FALSE);
}
}

argc -= optind;
argv += optind;

/* if argc == 0 we will execute zpool_sync_one on all pools */
ret = for_each_pool(argc, argv, B_FALSE, NULL, zpool_sync_one, &force);

return (ret);
}

typedef struct iostat_cbdata {
boolean_t cb_verbose;
int cb_namewidth;


+ 2
- 0
cddl/contrib/opensolaris/lib/libzfs/common/libzfs.h View File

@@ -271,6 +271,8 @@ extern int zpool_clear(zpool_handle_t *, const char *, nvlist_t *);
extern int zpool_reguid(zpool_handle_t *);
extern int zpool_reopen(zpool_handle_t *);

extern int zpool_sync_one(zpool_handle_t *, void *);

extern int zpool_vdev_online(zpool_handle_t *, const char *, int,
vdev_state_t *);
extern int zpool_vdev_offline(zpool_handle_t *, const char *, boolean_t);


+ 21
- 0
cddl/contrib/opensolaris/lib/libzfs/common/libzfs_pool.c View File

@@ -3515,6 +3515,27 @@ zpool_reopen(zpool_handle_t *zhp)
return (zpool_standard_error(hdl, errno, msg));
}

/* call into libzfs_core to execute the sync IOCTL per pool */
int
zpool_sync_one(zpool_handle_t *zhp, void *data)
{
int ret;
libzfs_handle_t *hdl = zpool_get_handle(zhp);
const char *pool_name = zpool_get_name(zhp);
boolean_t *force = data;
nvlist_t *innvl = fnvlist_alloc();

fnvlist_add_boolean_value(innvl, "force", *force);
if ((ret = lzc_sync(pool_name, innvl, NULL)) != 0) {
nvlist_free(innvl);
return (zpool_standard_error_fmt(hdl, ret,
dgettext(TEXT_DOMAIN, "sync '%s' failed"), pool_name));
}
nvlist_free(innvl);

return (0);
}

/*
* Convert from a devid string to a path.
*/


+ 23
- 11
cddl/contrib/opensolaris/lib/libzfs_core/common/libzfs_core.c View File

@@ -24,6 +24,7 @@
* Copyright (c) 2013 Steven Hartland. All rights reserved.
* Copyright (c) 2014 Integros [integros.com]
* Copyright 2017 RackTop Systems.
* Copyright (c) 2017 Datto Inc.
*/

/*
@@ -137,16 +138,17 @@ lzc_ioctl(zfs_ioc_t ioc, const char *name,
{
zfs_cmd_t zc = { 0 };
int error = 0;
char *packed;
char *packed = NULL;
#ifdef __FreeBSD__
nvlist_t *oldsource;
#endif
size_t size;
size_t size = 0;

ASSERT3S(g_refcount, >, 0);
VERIFY3S(g_fd, !=, -1);

(void) strlcpy(zc.zc_name, name, sizeof (zc.zc_name));
if (name != NULL)
(void) strlcpy(zc.zc_name, name, sizeof (zc.zc_name));

#ifdef __FreeBSD__
if (zfs_ioctl_version == ZFS_IOCVER_UNDEF)
@@ -160,9 +162,11 @@ lzc_ioctl(zfs_ioc_t ioc, const char *name,
}
#endif

packed = fnvlist_pack(source, &size);
zc.zc_nvlist_src = (uint64_t)(uintptr_t)packed;
zc.zc_nvlist_src_size = size;
if (source != NULL) {
packed = fnvlist_pack(source, &size);
zc.zc_nvlist_src = (uint64_t)(uintptr_t)packed;
zc.zc_nvlist_src_size = size;
}

if (resultp != NULL) {
*resultp = NULL;
@@ -462,6 +466,18 @@ lzc_exists(const char *dataset)
return (ioctl(g_fd, ZFS_IOC_OBJSET_STATS, &zc) == 0);
}

/*
* outnvl is unused.
* It was added to preserve the function signature in case it is
* needed in the future.
*/
/*ARGSUSED*/
int
lzc_sync(const char *pool_name, nvlist_t *innvl, nvlist_t **outnvl)
{
return (lzc_ioctl(ZFS_IOC_POOL_SYNC, pool_name, innvl, NULL));
}

/*
* Create "user holds" on snapshots. If there is a hold on a snapshot,
* the snapshot can not be destroyed. (However, it can be marked for deletion
@@ -562,11 +578,7 @@ lzc_release(nvlist_t *holds, nvlist_t **errlist)
int
lzc_get_holds(const char *snapname, nvlist_t **holdsp)
{
int error;
nvlist_t *innvl = fnvlist_alloc();
error = lzc_ioctl(ZFS_IOC_GET_HOLDS, snapname, innvl, holdsp);
fnvlist_free(innvl);
return (error);
return (lzc_ioctl(ZFS_IOC_GET_HOLDS, snapname, NULL, holdsp));
}

/*


+ 3
- 0
cddl/contrib/opensolaris/lib/libzfs_core/common/libzfs_core.h View File

@@ -23,6 +23,7 @@
* Copyright (c) 2012, 2016 by Delphix. All rights reserved.
* Copyright (c) 2013 by Martin Matuska <mm@FreeBSD.org>. All rights reserved.
* Copyright 2017 RackTop Systems.
* Copyright (c) 2017 Datto Inc.
*/

#ifndef _LIBZFS_CORE_H
@@ -91,6 +92,8 @@ boolean_t lzc_exists(const char *);
int lzc_rollback(const char *, char *, int);
int lzc_rollback_to(const char *, const char *);

int lzc_sync(const char *, nvlist_t *, nvlist_t **);

int lzc_rename(const char *, const char *);
int lzc_destroy(const char *);



+ 36
- 0
contrib/libc++/include/cmath View File

@@ -303,11 +303,15 @@ long double truncl(long double x);
#include <__config>
#include <math.h>
#include <version>
#include <type_traits>

#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
#pragma GCC system_header
#endif

_LIBCPP_PUSH_MACROS
#include <__undef_macros>

_LIBCPP_BEGIN_NAMESPACE_STD

using ::signbit;
@@ -632,6 +636,38 @@ lerp(long double __a, long double __b, long double __t) _NOEXCEPT { return __ler

#endif // _LIBCPP_STD_VER > 17

template <class _IntT, class _FloatT,
bool _FloatBigger = (numeric_limits<_FloatT>::digits > numeric_limits<_IntT>::digits),
int _Bits = (numeric_limits<_IntT>::digits - numeric_limits<_FloatT>::digits)>
_LIBCPP_INLINE_VISIBILITY
_LIBCPP_CONSTEXPR _IntT __max_representable_int_for_float() _NOEXCEPT {
static_assert(is_floating_point<_FloatT>::value, "must be a floating point type");
static_assert(is_integral<_IntT>::value, "must be an integral type");
static_assert(numeric_limits<_FloatT>::radix == 2, "FloatT has incorrect radix");
static_assert(_IsSame<_FloatT, float>::value || _IsSame<_FloatT, double>::value
|| _IsSame<_FloatT,long double>::value, "unsupported floating point type");
return _FloatBigger ? numeric_limits<_IntT>::max() : (numeric_limits<_IntT>::max() >> _Bits << _Bits);
}

// Convert a floating point number to the specified integral type after
// clamping to the integral types representable range.
//
// The behavior is undefined if `__r` is NaN.
template <class _IntT, class _RealT>
_LIBCPP_INLINE_VISIBILITY
_IntT __clamp_to_integral(_RealT __r) _NOEXCEPT {
using _Lim = std::numeric_limits<_IntT>;
const _IntT _MaxVal = std::__max_representable_int_for_float<_IntT, _RealT>();
if (__r >= ::nextafter(static_cast<_RealT>(_MaxVal), INFINITY)) {
return _Lim::max();
} else if (__r <= _Lim::lowest()) {
return _Lim::min();
}
return static_cast<_IntT>(__r);
}

_LIBCPP_END_NAMESPACE_STD

_LIBCPP_POP_MACROS

#endif // _LIBCPP_CMATH

+ 24
- 21
contrib/libc++/include/random View File

@@ -4592,7 +4592,10 @@ public:

template<class _IntType>
poisson_distribution<_IntType>::param_type::param_type(double __mean)
: __mean_(__mean)
// According to the standard `inf` is a valid input, but it causes the
// distribution to hang, so we replace it with the maximum representable
// mean.
: __mean_(isinf(__mean) ? numeric_limits<double>::max() : __mean)
{
if (__mean_ < 10)
{
@@ -4610,7 +4613,7 @@ poisson_distribution<_IntType>::param_type::param_type(double __mean)
{
__s_ = _VSTD::sqrt(__mean_);
__d_ = 6 * __mean_ * __mean_;
__l_ = static_cast<result_type>(__mean_ - 1.1484);
__l_ = std::trunc(__mean_ - 1.1484);
__omega_ = .3989423 / __s_;
double __b1_ = .4166667E-1 / __mean_;
double __b2_ = .3 * __b1_ * __b1_;
@@ -4627,12 +4630,12 @@ template<class _URNG>
_IntType
poisson_distribution<_IntType>::operator()(_URNG& __urng, const param_type& __pr)
{
result_type __x;
double __tx;
uniform_real_distribution<double> __urd;
if (__pr.__mean_ < 10)
{
__x = 0;
for (double __p = __urd(__urng); __p > __pr.__l_; ++__x)
__tx = 0;
for (double __p = __urd(__urng); __p > __pr.__l_; ++__tx)
__p *= __urd(__urng);
}
else
@@ -4642,19 +4645,19 @@ poisson_distribution<_IntType>::operator()(_URNG& __urng, const param_type& __pr
double __u;
if (__g > 0)
{
__x = static_cast<result_type>(__g);
if (__x >= __pr.__l_)
return __x;
__difmuk = __pr.__mean_ - __x;
__tx = std::trunc(__g);
if (__tx >= __pr.__l_)
return std::__clamp_to_integral<result_type>(__tx);
__difmuk = __pr.__mean_ - __tx;
__u = __urd(__urng);
if (__pr.__d_ * __u >= __difmuk * __difmuk * __difmuk)
return __x;
return std::__clamp_to_integral<result_type>(__tx);
}
exponential_distribution<double> __edist;
for (bool __using_exp_dist = false; true; __using_exp_dist = true)
{
double __e;
if (__using_exp_dist || __g < 0)
if (__using_exp_dist || __g <= 0)
{
double __t;
do
@@ -4664,31 +4667,31 @@ poisson_distribution<_IntType>::operator()(_URNG& __urng, const param_type& __pr
__u += __u - 1;
__t = 1.8 + (__u < 0 ? -__e : __e);
} while (__t <= -.6744);
__x = __pr.__mean_ + __pr.__s_ * __t;
__difmuk = __pr.__mean_ - __x;
__tx = std::trunc(__pr.__mean_ + __pr.__s_ * __t);
__difmuk = __pr.__mean_ - __tx;
__using_exp_dist = true;
}
double __px;
double __py;
if (__x < 10)
if (__tx < 10 && __tx >= 0)
{
const double __fac[] = {1, 1, 2, 6, 24, 120, 720, 5040,
40320, 362880};
__px = -__pr.__mean_;
__py = _VSTD::pow(__pr.__mean_, (double)__x) / __fac[__x];
__py = _VSTD::pow(__pr.__mean_, (double)__tx) / __fac[static_cast<int>(__tx)];
}
else
{
double __del = .8333333E-1 / __x;
double __del = .8333333E-1 / __tx;
__del -= 4.8 * __del * __del * __del;
double __v = __difmuk / __x;
double __v = __difmuk / __tx;
if (_VSTD::abs(__v) > 0.25)
__px = __x * _VSTD::log(1 + __v) - __difmuk - __del;
__px = __tx * _VSTD::log(1 + __v) - __difmuk - __del;
else
__px = __x * __v * __v * (((((((.1250060 * __v + -.1384794) *
__px = __tx * __v * __v * (((((((.1250060 * __v + -.1384794) *
__v + .1421878) * __v + -.1661269) * __v + .2000118) *
__v + -.2500068) * __v + .3333333) * __v + -.5) - __del;
__py = .3989423 / _VSTD::sqrt(__x);
__py = .3989423 / _VSTD::sqrt(__tx);
}
double __r = (0.5 - __difmuk) / __pr.__s_;
double __r2 = __r * __r;
@@ -4708,7 +4711,7 @@ poisson_distribution<_IntType>::operator()(_URNG& __urng, const param_type& __pr
}
}
}
return __x;
return std::__clamp_to_integral<result_type>(__tx);
}

template <class _CharT, class _Traits, class _IntType>


+ 12
- 11
contrib/libxo/Makefile.am View File

@@ -32,7 +32,6 @@ errors:
docs:
@(cd doc ; ${MAKE} docs)


DIST_FILES_DIR = ~/Dropbox/dist-files/
GH_PAGES_DIR = gh-pages/
GH_PAGES_DIR_VER = gh-pages/${PACKAGE_VERSION}
@@ -49,18 +48,20 @@ upload: dist upload-docs upload-xohtml-files
@echo "Remember to run:"
@echo " gt tag ${PACKAGE_VERSION}"

upload-docs: docs
@echo "Uploading libxo-manual.html ... "
@-[ -d ${GH_PAGES_DIR} ] \
&& echo "Updating manual on gh-pages ..." \
&& mkdir -p ${GH_PAGES_DIR_VER} \
&& cp doc/libxo-manual.html ${GH_PAGES_DIR} \
&& cp doc/libxo-manual.html ${GH_PAGES_DIR_VER} \
upload-docs: docs upload-html

upload-html:
@echo "Uploading html ... "
@-[ -d ${GH_PAGES_DIR} -a -d doc/html ] \
&& echo "Updating html on gh-pages ..." \
&& mkdir -p ${GH_PAGES_DIR_VER}/html \
&& cp doc/top-link.html ${GH_PAGES_DIR}/libxo.html \
&& cp -r doc/html/* ${GH_PAGES_DIR_VER}/html/ \
&& (cd ${GH_PAGES_DIR} \
&& git add ${PACKAGE_VERSION} \
&& git add libxo-manual.html \
&& git add libxo.html \
&& git add ${PACKAGE_VERSION}/html \
&& git commit -m 'new docs' \
libxo-manual.html ${PACKAGE_VERSION} \
libxo.html ${PACKAGE_VERSION}/html \
&& git push origin gh-pages ) ; true

upload-xohtml-files:


+ 3
- 0
contrib/libxo/README.md View File

@@ -10,6 +10,9 @@ application calls a function "xo_emit" to product output that is
described in a format string. A "field descriptor" tells libxo what
the field is and what it means.

Imagine a simplified ``wc`` that emits its output fields in a single
xo_emit call:

```
xo_emit(" {:lines/%7ju/%ju} {:words/%7ju/%ju} "
"{:characters/%7ju/%ju}{d:filename/%s}\n",


+ 3
- 1
contrib/libxo/configure.ac View File

@@ -12,7 +12,7 @@
#

AC_PREREQ(2.2)
AC_INIT([libxo], [1.0.4], [phil@juniper.net])
AC_INIT([libxo], [1.3.1], [phil@juniper.net])
AM_INIT_AUTOMAKE([-Wall -Werror foreign -Wno-portability])

# Support silent build rules. Requires at least automake-1.11.
@@ -452,6 +452,7 @@ AC_CONFIG_FILES([
libxo/add.man
encoder/Makefile
encoder/cbor/Makefile
encoder/csv/Makefile
encoder/test/Makefile
xo/Makefile
xolint/Makefile
@@ -459,6 +460,7 @@ AC_CONFIG_FILES([
xopo/Makefile
packaging/libxo.pc
doc/Makefile
doc/top-link.html
tests/Makefile
tests/core/Makefile
tests/gettext/Makefile


+ 14
- 60
contrib/libxo/doc/Makefile.am View File

@@ -8,68 +8,22 @@
# using the SOFTWARE, you agree to be bound by the terms of that
# LICENSE.

if HAVE_OXTRADOC
OXTRADOC_DIR = ${SLAX_OXTRADOCDIR}
OXTRADOC_PREFIX = ${OXTRADOC_DIR}
OXTRADOC = ${OXTRADOC_DIR}/oxtradoc
SLAXPROC_BINDIR = ${SLAX_BINDIR}
doc docs: xolint.rst html

XML2RFC = ${OXTRADOC_DIR}/xml2rfc.tcl
XML2HTMLDIR = ${OXTRADOC_DIR}
XML2HTMLBIN = ${XML2HTMLDIR}/rfc2629-to-html.slax
SLAXPROC = ${SLAX_BINDIR}/slaxproc

SLAXPROC_ARGS = \
-a oxtradoc-dir ${OXTRADOC_DIR} \
-a oxtradoc-install-dir ${OXTRADOC_DIR} \
-a anchor-prefix docs

SLAXPROC_ARGS_INLINE = \
-a oxtradoc-inline yes

SLAXPROC_ARGS += ${SLAXPROC_ARGS_INLINE}

XML2HTML = \
${SLAXPROC} -g -e -I ${OXTRADOC_DIR} -I . \
${SLAXPROC_ARGS} \
${XML2HTMLBIN}

OX_ARGS = -P ${OXTRADOC_PREFIX} -L ${OXTRADOC_PREFIX}
OX_ARGS += -S ${SLAXPROC} -p doc
OX_CMD = ${PERL} ${PERLOPTS} ${OXTRADOC} ${OX_ARGS}
OXTRADOC_CMD = ${OX_CMD}

OUTPUT = libxo-manual
INPUT = libxo

EXTRA_DIST = \
${INPUT}.txt \
${OUTPUT}.html \
${OUTPUT}.txt

doc docs: ${OUTPUT}.txt ${OUTPUT}.html

${OUTPUT}.txt: ${INPUT}.txt ${OXTRADOC} xolint.txt
${OXTRADOC_CMD} -m text -o $@ $<

${OUTPUT}.html: ${INPUT}.txt ${OXTRADOC} ${XML2HTMLBIN} xolint.txt
${OXTRADOC_CMD} -m html -o $@ $<

xolint.txt: ${top_srcdir}/xolint/xolint.pl
perl ${top_srcdir}/xolint/xolint.pl -D > xolint.txt
#
# The contents of xolint.rst is generated based on xolint.pl, since we
# really want this to be self-documenting. But readthedocs.org needs this
# data to be _in_ repo. So we generate this file on command only, and
# the developer needs to commit any changes.
#

CLEANFILES = \
xolint.txt \
${INPUT}.xml \
${INPUT}.fxml \
${OUTPUT}.txt \
${OUTPUT}.html
else
doc docs:
@${ECHO} "The 'oxtradoc' tool is not installed; see libslax.org"
endif
xolint.rst: ${top_srcdir}/xolint/xolint.pl
perl ${top_srcdir}/xolint/xolint.pl -D > ${top_srcdir}/doc/xolint.rst

SPHINX = python3.4 -msphinx
SPHINX = python3 -msphinx

html sphinx sphinx-html:
${SPHINX} -M html ${srcdir} .
${SPHINX} -M html ${srcdir} . -N -E

singlehtml:
${SPHINX} -M singlehtml ${srcdir} . -N -E

+ 98
- 98
contrib/libxo/doc/api.rst View File

@@ -1,4 +1,4 @@
.. index: API
.. index:: API

The libxo API
=============
@@ -155,14 +155,14 @@ Output Styles (XO_STYLE\_\*)

The libxo functions accept a set of output styles:

=============== =========================
Flag Description
=============== =========================
XO_STYLE_TEXT Traditional text output
XO_STYLE_XML XML encoded data
XO_STYLE_JSON JSON encoded data
XO_STYLE_HTML HTML encoded data
=============== =========================
=============== =========================
Flag Description
=============== =========================
XO_STYLE_TEXT Traditional text output
XO_STYLE_XML XML encoded data
XO_STYLE_JSON JSON encoded data
XO_STYLE_HTML HTML encoded data
=============== =========================

The "XML", "JSON", and "HTML" output styles all use the UTF-8
character encoding. "TEXT" using locale-based encoding.
@@ -256,26 +256,26 @@ Flags (XOF\_\*)

The set of valid flags include:

=================== =========================================
Flag Description
=================== =========================================
XOF_CLOSE_FP Close file pointer on `xo_destroy`
XOF_COLOR Enable color and effects in output
XOF_COLOR_ALLOWED Allow color/effect for terminal output
XOF_DTRT Enable "do the right thing" mode
XOF_INFO Display info data attributes (HTML)
XOF_KEYS Emit the key attribute (XML)
XOF_NO_ENV Do not use the :ref:`libxo-options` env var
XOF_NO_HUMANIZE Display humanization (TEXT, HTML)
XOF_PRETTY Make "pretty printed" output
XOF_UNDERSCORES Replaces hyphens with underscores
XOF_UNITS Display units (XML, HMTL)
XOF_WARN Generate warnings for broken calls
XOF_WARN_XML Generate warnings in XML on stdout
XOF_XPATH Emit XPath expressions (HTML)
XOF_COLUMNS Force xo_emit to return columns used
XOF_FLUSH Flush output after each `xo_emit` call
=================== =========================================
=================== =========================================
Flag Description
=================== =========================================
XOF_CLOSE_FP Close file pointer on `xo_destroy`
XOF_COLOR Enable color and effects in output
XOF_COLOR_ALLOWED Allow color/effect for terminal output
XOF_DTRT Enable "do the right thing" mode
XOF_INFO Display info data attributes (HTML)
XOF_KEYS Emit the key attribute (XML)
XOF_NO_ENV Do not use the :ref:`libxo-options` env var
XOF_NO_HUMANIZE Display humanization (TEXT, HTML)
XOF_PRETTY Make "pretty printed" output
XOF_UNDERSCORES Replaces hyphens with underscores
XOF_UNITS Display units (XML, HMTL)
XOF_WARN Generate warnings for broken calls
XOF_WARN_XML Generate warnings in XML on stdout
XOF_XPATH Emit XPath expressions (HTML)
XOF_COLUMNS Force xo_emit to return columns used
XOF_FLUSH Flush output after each `xo_emit` call
=================== =========================================

The `XOF_CLOSE_FP` flag will trigger the call of the *close_func*
(provided via `xo_set_writer`) when the handle is destroyed.
@@ -300,12 +300,12 @@ regardless of whether warnings are enabled.
If the style is `XO_STYLE_HTML`, the following additional flags can be
used:

=============== =========================================
Flag Description
=============== =========================================
XOF_XPATH Emit "data-xpath" attributes
XOF_INFO Emit additional info fields
=============== =========================================
=============== =========================================
Flag Description
=============== =========================================
XOF_XPATH Emit "data-xpath" attributes
XOF_INFO Emit additional info fields
=============== =========================================

The `XOF_XPATH` flag enables the emission of XPath expressions detailing
the hierarchy of XML elements used to encode the data field, if the
@@ -317,11 +317,11 @@ output. See :ref:`field-information` for details.
If the style is `XO_STYLE_XML`, the following additional flags can be
used:

=============== =========================================
Flag Description
=============== =========================================
XOF_KEYS Flag "key" fields for XML
=============== =========================================
=============== =========================================
Flag Description
=============== =========================================
XOF_KEYS Flag "key" fields for XML
=============== =========================================

The `XOF_KEYS` flag adds "key" attribute to the XML encoding for
field definitions that use the "k" modifier. The key attribute has
@@ -1308,52 +1308,52 @@ These values are defined in <syslog.h>.
The priority value indicates the importance and potential impact of
each message:

============= =======================================================
Priority Description
============= =======================================================
LOG_EMERG A panic condition, normally broadcast to all users
LOG_ALERT A condition that should be corrected immediately
LOG_CRIT Critical conditions
LOG_ERR Generic errors
LOG_WARNING Warning messages
LOG_NOTICE Non-error conditions that might need special handling
LOG_INFO Informational messages
LOG_DEBUG Developer-oriented messages
============= =======================================================
============= =======================================================
Priority Description
============= =======================================================
LOG_EMERG A panic condition, normally broadcast to all users
LOG_ALERT A condition that should be corrected immediately
LOG_CRIT Critical conditions
LOG_ERR Generic errors
LOG_WARNING Warning messages
LOG_NOTICE Non-error conditions that might need special handling
LOG_INFO Informational messages
LOG_DEBUG Developer-oriented messages
============= =======================================================

The facility value indicates the source of message, in fairly generic
terms:

=============== =======================================================
Facility Description
=============== =======================================================
LOG_AUTH The authorization system (e.g. :manpage:`login(1)`)
LOG_AUTHPRIV As LOG_AUTH, but logged to a privileged file
LOG_CRON The cron daemon: :manpage:`cron(8)`
LOG_DAEMON System daemons, not otherwise explicitly listed
LOG_FTP The file transfer protocol daemons
LOG_KERN Messages generated by the kernel
LOG_LPR The line printer spooling system
LOG_MAIL The mail system
LOG_NEWS The network news system
LOG_SECURITY Security subsystems, such as :manpage:`ipfw(4)`
LOG_SYSLOG Messages generated internally by :manpage:`syslogd(8)`
LOG_USER Messages generated by user processes (default)
LOG_UUCP The uucp system
LOG_LOCAL0..7 Reserved for local use
=============== =======================================================
=============== =======================================================
Facility Description
=============== =======================================================
LOG_AUTH The authorization system (e.g. :manpage:`login(1)`)
LOG_AUTHPRIV As LOG_AUTH, but logged to a privileged file
LOG_CRON The cron daemon: :manpage:`cron(8)`
LOG_DAEMON System daemons, not otherwise explicitly listed
LOG_FTP The file transfer protocol daemons
LOG_KERN Messages generated by the kernel
LOG_LPR The line printer spooling system
LOG_MAIL The mail system
LOG_NEWS The network news system
LOG_SECURITY Security subsystems, such as :manpage:`ipfw(4)`
LOG_SYSLOG Messages generated internally by :manpage:`syslogd(8)`
LOG_USER Messages generated by user processes (default)
LOG_UUCP The uucp system
LOG_LOCAL0..7 Reserved for local use
=============== =======================================================

In addition to the values listed above, xo_open_log accepts a set of
addition flags requesting specific logging behaviors:

============ ====================================================
Flag Description
============ ====================================================
LOG_CONS If syslogd fails, attempt to write to /dev/console
LOG_NDELAY Open the connection to :manpage:`syslogd(8)` immediately
LOG_PERROR Write the message also to standard error output
LOG_PID Log the process id with each message
============ ====================================================
============ ====================================================
Flag Description
============ ====================================================
LOG_CONS If syslogd fails, attempt to write to /dev/console
LOG_NDELAY Open the connection to :manpage:`syslogd(8)` immediately
LOG_PERROR Write the message also to standard error output
LOG_PID Log the process id with each message
============ ====================================================

.. index:: xo_syslog

@@ -1588,26 +1588,26 @@ processing model of libxo. Content is formatted within libxo, and
callbacks are made to the encoder's handler function when data is
ready to be processed:

======================= =======================================
Operation Meaning (Base function)
======================= =======================================
XO_OP_CREATE Called when the handle is created
XO_OP_OPEN_CONTAINER Container opened (xo_open_container)
XO_OP_CLOSE_CONTAINER Container closed (xo_close_container)
XO_OP_OPEN_LIST List opened (xo_open_list)
XO_OP_CLOSE_LIST List closed (xo_close_list)
XO_OP_OPEN_LEAF_LIST Leaf list opened (xo_open_leaf_list)
XO_OP_CLOSE_LEAF_LIST Leaf list closed (xo_close_leaf_list)
XO_OP_OPEN_INSTANCE Instance opened (xo_open_instance)
XO_OP_CLOSE_INSTANCE Instance closed (xo_close_instance)
XO_OP_STRING Field with Quoted UTF-8 string
XO_OP_CONTENT Field with content
XO_OP_FINISH Finish any pending output
XO_OP_FLUSH Flush any buffered output
XO_OP_DESTROY Clean up resources
XO_OP_ATTRIBUTE An attribute name/value pair
XO_OP_VERSION A version string
======================= =======================================
======================= =======================================
Operation Meaning (Base function)
======================= =======================================
XO_OP_CREATE Called when the handle is created
XO_OP_OPEN_CONTAINER Container opened (xo_open_container)
XO_OP_CLOSE_CONTAINER Container closed (xo_close_container)
XO_OP_OPEN_LIST List opened (xo_open_list)
XO_OP_CLOSE_LIST List closed (xo_close_list)
XO_OP_OPEN_LEAF_LIST Leaf list opened (xo_open_leaf_list)
XO_OP_CLOSE_LEAF_LIST Leaf list closed (xo_close_leaf_list)
XO_OP_OPEN_INSTANCE Instance opened (xo_open_instance)
XO_OP_CLOSE_INSTANCE Instance closed (xo_close_instance)
XO_OP_STRING Field with Quoted UTF-8 string
XO_OP_CONTENT Field with content
XO_OP_FINISH Finish any pending output
XO_OP_FLUSH Flush any buffered output
XO_OP_DESTROY Clean up resources
XO_OP_ATTRIBUTE An attribute name/value pair
XO_OP_VERSION A version string
======================= =======================================

For all the open and close operations, the name parameter holds the
name of the construct. For string, content, and attribute operations,


+ 11
- 3
contrib/libxo/doc/conf.py View File

@@ -21,6 +21,14 @@
# import sys
# sys.path.insert(0, os.path.abspath('.'))

import subprocess

#
# Instead of hardcoding the version number here, we read it from the
# project's configure script
#
vers_cmd = "grep AC_INIT ../configure.ac | awk '{ print substr($2, 2, length($2) - 3);}'"
version = subprocess.check_output(vers_cmd, shell=True).decode("utf-8")

# -- General configuration ------------------------------------------------

@@ -47,7 +55,7 @@ master_doc = 'index'

# General information about the project.
project = 'libxo'
copyright = '2017, Juniper Networks'
copyright = '2017-2019, Juniper Networks Inc'
author = 'Phil Shafer'
default_role = 'code'
primary_domain = 'c'
@@ -58,9 +66,9 @@ smart_quotes = False
# built documents.
#
# The short X.Y version.
version = '0.8.4'
#version = 'develop'
# The full version, including alpha/beta/rc tags.
release = '0.8.4'
release = version

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.


+ 269
- 0
contrib/libxo/doc/encoders.rst View File

@@ -0,0 +1,269 @@
.. index:: encoder

Encoders
========

This section gives an overview of encoders, details on the encoders
that ship with libxo, and documentation for developers of future
encoders.

Overview
--------

The libxo library contains software to generate four "built-in"
formats: text, XML, JSON, and HTML. These formats are common and
useful, but there are other common and useful formats that users will
want, and including them all in the libxo software would be difficult
and cumbersome.

To allow support for additional encodings, libxo includes a
"pluggable" extension mechanism for dynamically loading new encoders.
libxo-based applications can automatically use any installed encoder.

Use the "encoder=XXX" option to access encoders. The following
example uses the "cbor" encoder, saving the output into a file::

df --libxo encoder=cbor > df-output.cbor

Encoders can support specific options that can be accessed by
following the encoder name with a colon (':') and one of more options,
separated by a plus sign "+"::

df --libxo encoder=csv:path=filesystem+leaf=name+no-header

This example instructs libxo to load the "csv" encoder and pass the
following options::

path=filesystem
leaf=name
no-header

Each of these option is interpreted by the encoder, and all such
options names and semantics are specific to the particular encoder.
Refer to the intended encoder for documentation on its options.

.. _csv_encoder:

CSV - Comma Separated Values
----------------------------

libxo ships with a custom encoder for "CSV" files, a common format for
comma separated values. The output of the CSV encoder can be loaded
directly into spreadsheets or similar applications.

A standard for CSV files is provided in :RFC:`4180`, but since the
format predates that standard by decades, there are many minor
differences in CSV file consumers and their expectations. The CSV
encoder has a number of options to tailor output to those
expectations.

Consider the following XML::

% list-items --libxo xml,pretty
<top>
<data test="value">
<item test2="value2">
<sku test3="value3" key="key">GRO-000-415</sku>
<name key="key">gum</name>
<sold>1412</sold>
<in-stock>54</in-stock>
<on-order>10</on-order>
</item>
<item>
<sku test3="value3" key="key">HRD-000-212</sku>
<name key="key">rope</name>
<sold>85</sold>
<in-stock>4</in-stock>
<on-order>2</on-order>
</item>
<item>
<sku test3="value3" key="key">HRD-000-517</sku>
<name key="key">ladder</name>
<sold>0</sold>
<in-stock>2</in-stock>
<on-order>1</on-order>
</item>
</data>
</top>

This output is a list of `instances` (named "item"), each containing a
set of `leafs` ("sku", "name", etc).

The CSV encoder will emit the leaf values in this output as `fields`
inside a CSV `record`, which is a line containing a set of
comma-separated values::

% list-items --libxo encoder=csv
sku,name,sold,in-stock,on-order
GRO-000-415,gum,1412,54,10
HRD-000-212,rope,85,4,2
HRD-000-517,ladder,0,2,1

Be aware that since the CSV encoder looks for data instances, when
used with :ref:`xo`, the `--instance` option will be needed::

% xo --libxo encoder=csv --instance foo 'The {:product} is {:status}\n' stereo "in route"
product,status
stereo,in route

.. _csv_path:

The `path` Option
~~~~~~~~~~~~~~~~~

By default, the CSV encoder will attempt to emit any list instance
generated by the application. In some cases, this may be
unacceptable, and a specific list may be desired.

Use the "path" option to limit the processing of output to a specific
hierarchy. The path should be one or more names of containers or
lists.

For example, if the "list-items" application generates other lists,
the user can give "path=top/data/item" as a path::

% list-items --libxo encoder=csv:path=top/data/item
sku,name,sold,in-stock,on-order
GRO-000-415,gum,1412,54,10
HRD-000-212,rope,85,4,2
HRD-000-517,ladder,0,2,1

Paths are "relative", meaning they need not be a complete set
of names to the list. This means that "path=item" may be sufficient
for the above example.

.. _csv_leafs:

The `leafs` Option
~~~~~~~~~~~~~~~~~~

The CSV encoding requires that all lines of output have the same
number of fields with the same order. In contrast, XML and JSON allow
any order (though libxo forces key leafs to appear before other
leafs).

To maintain a consistent set of fields inside the CSV file, the same
set of leafs must be selected from each list item. By default, the
CSV encoder records the set of leafs that appear in the first list
instance it processes, and extract only those leafs from future
instances. If the first instance is missing a leaf that is desired by
the consumer, the "leaf" option can be used to ensure that an empty
value is recorded for instances that lack a particular leaf.

The "leafs" option can also be used to exclude leafs, limiting the
output to only those leafs provided.

In addition, the order of the output fields follows the order in which
the leafs are listed. "leafs=one.two" and "leafs=two.one" give
distinct output.

So the "leafs" option can be used to expand, limit, and order the set
of leafs.

The value of the leafs option should be one or more leaf names,
separated by a period (".")::

% list-items --libxo encoder=csv:leafs=sku.on-order
sku,on-order
GRO-000-415,10
HRD-000-212,2
HRD-000-517,1
% list-items -libxo encoder=csv:leafs=on-order.sku
on-order,sku
10,GRO-000-415
2,HRD-000-212
1,HRD-000-517

Note that since libxo uses terminology from YANG (:RFC:`7950`), the
data modeling language for NETCONF (:RFC:`6241`), which uses "leafs"
as the plural form of "leaf". libxo follows that convention.

.. _csv_no_header:

The `no-header` Option
~~~~~~~~~~~~~~~~~~~~~~

CSV files typical begin with a line that defines the fields included
in that file, in an attempt to make the contents self-defining::

sku,name,sold,in-stock,on-order
GRO-000-415,gum,1412,54,10
HRD-000-212,rope,85,4,2
HRD-000-517,ladder,0,2,1

There is no reliable mechanism for determining whether this header
line is included, so the consumer must make an assumption.

The csv encoder defaults to producing the header line, but the
"no-header" option can be included to avoid the header line.

.. _csv_no_quotes:

The `no-quotes` Option
~~~~~~~~~~~~~~~~~~~~~~

:RFC:`4180` specifies that fields containing spaces should be quoted, but
many CSV consumers do not handle quotes. The "no-quotes" option
instruct the CSV encoder to avoid the use of quotes.

.. _csv_dos:

The `dos` Option
~~~~~~~~~~~~~~~~

:RFC:`4180` defines the end-of-line marker as a carriage return
followed by a newline. This `CRLF` convention dates from the distant
past, but its use was anchored in the 1980s by the `DOS` operating
system.

The CSV encoder defaults to using the standard Unix end-of-line
marker, a simple newline. Use the "dos" option to use the `CRLF`
convention.

The Encoder API
---------------

The encoder API consists of three distinct phases:

- loading the encoder
- initializing the encoder
- feeding operations to the encoder

To load the encoder, libxo will open a shared library named:

${prefix}/lib/libxo/encoder/${name}.enc

This file is typically a symbolic link to a dynamic library, suitable
for `dlopen`(). libxo looks for a symbol called
`xo_encoder_library_init` inside that library and calls it with the
arguments defined in the header file "xo_encoder.h". This function
should look as follows::

int
xo_encoder_library_init (XO_ENCODER_INIT_ARGS)
{
arg->xei_version = XO_ENCODER_VERSION;
arg->xei_handler = test_handler;
return 0;
}

Several features here allow for future compatibility: the macro
XO_ENCODER_INIT_ARGS allows the arguments to this function change over
time, and the XO_ENCODER_VERSION allows the library to tell libxo
which version of the API it was compiled with.

The function places in xei_handler should be have the signature::

static int
test_handler (XO_ENCODER_HANDLER_ARGS)
{
...

This function will be called with the "op" codes defined in
"xo_encoder.h". Each op code represents a distinct event in the libxo
processing model. For example OP_OPEN_CONTAINER tells the encoder
that a new container has been opened, and the encoder can behave in an
appropriate manner.



+ 3
- 4
contrib/libxo/doc/faq.rst View File

@@ -202,8 +202,7 @@ will lead users to ask the difference between the two fields. If
there is no difference, use only one of the field names. If there is
a difference, change the names to make that difference more obvious.

.. ignore for now, since we want can't have generated content
What does this message mean?
----------------------------
What does this message mean?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

!!include-file xolint.txt
.. include:: xolint.rst

+ 71
- 71
contrib/libxo/doc/field-formatting.rst View File

@@ -53,48 +53,48 @@ removed eventually.

The format character is described in the following table:

===== ================= ======================
Ltr Argument Type Format
===== ================= ======================
d int base 10 (decimal)
i int base 10 (decimal)
o int base 8 (octal)
u unsigned base 10 (decimal)
x unsigned base 16 (hex)
X unsigned long base 16 (hex)
D long base 10 (decimal)
O unsigned long base 8 (octal)
U unsigned long base 10 (decimal)
e double [-]d.ddde+-dd
E double [-]d.dddE+-dd
f double [-]ddd.ddd
F double [-]ddd.ddd
g double as 'e' or 'f'
G double as 'E' or 'F'
a double [-]0xh.hhhp[+-]d
A double [-]0Xh.hhhp[+-]d
c unsigned char a character
C wint_t a character
s char \* a UTF-8 string
S wchar_t \* a unicode/WCS string
p void \* '%#lx'
===== ================= ======================
===== ================= ======================
Ltr Argument Type Format
===== ================= ======================
d int base 10 (decimal)
i int base 10 (decimal)
o int base 8 (octal)
u unsigned base 10 (decimal)
x unsigned base 16 (hex)
X unsigned long base 16 (hex)
D long base 10 (decimal)
O unsigned long base 8 (octal)
U unsigned long base 10 (decimal)
e double [-]d.ddde+-dd
E double [-]d.dddE+-dd
f double [-]ddd.ddd
F double [-]ddd.ddd
g double as 'e' or 'f'
G double as 'E' or 'F'
a double [-]0xh.hhhp[+-]d
A double [-]0Xh.hhhp[+-]d
c unsigned char a character
C wint_t a character
s char \* a UTF-8 string
S wchar_t \* a unicode/WCS string
p void \* '%#lx'
===== ================= ======================

The 'h' and 'l' modifiers affect the size and treatment of the
argument:

===== ============= ====================
Mod d, i o, u, x, X
===== ============= ====================
hh signed char unsigned char
h short unsigned short
l long unsigned long
ll long long unsigned long long
j intmax_t uintmax_t
t ptrdiff_t ptrdiff_t
z size_t size_t
q quad_t u_quad_t
===== ============= ====================
===== ============= ====================
Mod d, i o, u, x, X
===== ============= ====================
hh signed char unsigned char
h short unsigned short
l long unsigned long
ll long long unsigned long long
j intmax_t uintmax_t
t ptrdiff_t ptrdiff_t
z size_t size_t
q quad_t u_quad_t
===== ============= ====================

.. index:: UTF-8
.. index:: Locale
@@ -122,14 +122,14 @@ style::
xo_emit("All strings are utf-8 content {:tag/%ls}",
L"except for wide strings");

======== ================== ===============================
Format Argument Type Argument Contents
======== ================== ===============================
%s const char \* UTF-8 string
%S const char \* UTF-8 string (alias for '%ls')
%ls const wchar_t \* Wide character UNICODE string
%hs const char * locale-based string
======== ================== ===============================
======== ================== ===============================
Format Argument Type Argument Contents
======== ================== ===============================
%s const char \* UTF-8 string
%S const char \* UTF-8 string (alias for '%ls')
%ls const wchar_t \* Wide character UNICODE string
%hs const char * locale-based string
======== ================== ===============================

.. admonition:: "Long", not "locale"

@@ -266,21 +266,21 @@ incompatible with printf-like testing:
If none of these features are in use by your code, then using the "_p"
variants might be wise:

================== ========================
Function printf-like Equivalent
================== ========================
xo_emit_hv xo_emit_hvp
xo_emit_h xo_emit_hp
xo_emit xo_emit_p
xo_emit_warn_hcv xo_emit_warn_hcvp
xo_emit_warn_hc xo_emit_warn_hcp
xo_emit_warn_c xo_emit_warn_cp
xo_emit_warn xo_emit_warn_p
xo_emit_warnx xo_emit_warnx_p
xo_emit_err xo_emit_err_p
xo_emit_errx xo_emit_errx_p
xo_emit_errc xo_emit_errc_p
================== ========================
================== ========================
Function printf-like Equivalent
================== ========================
xo_emit_hv xo_emit_hvp
xo_emit_h xo_emit_hp
xo_emit xo_emit_p
xo_emit_warn_hcv xo_emit_warn_hcvp
xo_emit_warn_hc xo_emit_warn_hcp
xo_emit_warn_c xo_emit_warn_cp
xo_emit_warn xo_emit_warn_p
xo_emit_warnx xo_emit_warnx_p
xo_emit_err xo_emit_err_p
xo_emit_errx xo_emit_errx_p
xo_emit_errc xo_emit_errc_p
================== ========================

.. index:: performance
.. index:: XOEF_RETAIN
@@ -305,16 +305,16 @@ xo_emit_f() function. A complete set of xo_emit_f functions exist to
match all the xo_emit function signatures (with handles, varadic
argument, and printf-like flags):

================== ========================
Function Flags Equivalent
================== ========================
xo_emit_hv xo_emit_hvf
xo_emit_h xo_emit_hf
xo_emit xo_emit_f
xo_emit_hvp xo_emit_hvfp
xo_emit_hp xo_emit_hfp
xo_emit_p xo_emit_fp
================== ========================
================== ========================
Function Flags Equivalent
================== ========================
xo_emit_hv xo_emit_hvf
xo_emit_h xo_emit_hf
xo_emit xo_emit_f
xo_emit_hvp xo_emit_hvfp
xo_emit_hp xo_emit_hfp
xo_emit_p xo_emit_fp
================== ========================

The format string must be immutable across multiple calls to xo_emit_f(),
since the library retains the string. Typically this is done by using


+ 20
- 20
contrib/libxo/doc/field-modifiers.rst View File

@@ -8,26 +8,26 @@ Field Modifiers
Field modifiers are flags which modify the way content emitted for
particular output styles:

=== =============== ===================================================
M Name Description
=== =============== ===================================================
a argument The content appears as a 'const char \*' argument
c colon A colon (":") is appended after the label
d display Only emit field for display styles (text/HTML)
e encoding Only emit for encoding styles (XML/JSON)
g gettext Call gettext on field's render content
h humanize (hn) Format large numbers in human-readable style
\ hn-space Humanize: Place space between numeric and unit
\ hn-decimal Humanize: Add a decimal digit, if number < 10
\ hn-1000 Humanize: Use 1000 as divisor instead of 1024
k key Field is a key, suitable for XPath predicates
l leaf-list Field is a leaf-list
n no-quotes Do not quote the field when using JSON style
p plural Gettext: Use comma-separated plural form
q quotes Quote the field when using JSON style
t trim Trim leading and trailing whitespace
w white A blank (" ") is appended after the label
=== =============== ===================================================
=== =============== ===================================================
M Name Description
=== =============== ===================================================
a argument The content appears as a 'const char \*' argument
c colon A colon (":") is appended after the label
d display Only emit field for display styles (text/HTML)
e encoding Only emit for encoding styles (XML/JSON)
g gettext Call gettext on field's render content
h humanize (hn) Format large numbers in human-readable style
\ hn-space Humanize: Place space between numeric and unit
\ hn-decimal Humanize: Add a decimal digit, if number < 10
\ hn-1000 Humanize: Use 1000 as divisor instead of 1024
k key Field is a key, suitable for XPath predicates
l leaf-list Field is a leaf-list
n no-quotes Do not quote the field when using JSON style
p plural Gettext: Use comma-separated plural form
q quotes Quote the field when using JSON style
t trim Trim leading and trailing whitespace
w white A blank (" ") is appended after the label
=== =============== ===================================================

Roles and modifiers can also use more verbose names, when preceded by
a comma. For example, the modifier string "Lwc" (or "L,white,colon")


+ 46
- 44
contrib/libxo/doc/field-roles.rst View File

@@ -8,23 +8,25 @@ Field Roles
Field roles are optional, and indicate the role and formatting of the
content. The roles are listed below; only one role is permitted:

=== ============== =================================================
R Name Description
=== ============== =================================================
C color Field has color and effect controls
D decoration Field is non-text (e.g., colon, comma)
E error Field is an error message
G gettext Call gettext(3) on the format string
L label Field is text that prefixes a value
N note Field is text that follows a value
P padding Field is spaces needed for vertical alignment
T title Field is a title value for headings
U units Field is the units for the previous value field
V value Field is the name of field (the default)
W warning Field is a warning message
[ start-anchor Begin a section of anchored variable-width text
] stop-anchor End a section of anchored variable-width text
=== ============== =================================================
=== ============== =================================================
R Name Description
=== ============== =================================================
C color Field has color and effect controls
D decoration Field is non-text (e.g., colon, comma)
E error Field is an error message
G gettext Call gettext(3) on the format string
L label Field is text that prefixes a value
N note Field is text that follows a value
P padding Field is spaces needed for vertical alignment
T title Field is a title value for headings
U units Field is the units for the previous value field
V value Field is the name of field (the default)
W warning Field is a warning message
[ start-anchor Begin a section of anchored variable-width text
] stop-anchor End a section of anchored variable-width text
=== ============== =================================================

::

EXAMPLE:
xo_emit("{L:Free}{D::}{P: }{:free/%u} {U:Blocks}\n",
@@ -80,36 +82,36 @@ foreground and background colors, respectively::

The following table lists the supported effects:

=============== =================================================
Name Description
=============== =================================================
bg-XXXXX Change background color
bold Start bold text effect
fg-XXXXX Change foreground color
inverse Start inverse (aka reverse) text effect
no-bold Stop bold text effect
no-inverse Stop inverse (aka reverse) text effect
no-underline Stop underline text effect
normal Reset effects (only)
reset Reset colors and effects (restore defaults)
underline Start underline text effect
=============== =================================================
=============== =================================================
Name Description
=============== =================================================
bg-XXXXX Change background color
bold Start bold text effect
fg-XXXXX Change foreground color
inverse Start inverse (aka reverse) text effect
no-bold Stop bold text effect
no-inverse Stop inverse (aka reverse) text effect
no-underline Stop underline text effect
normal Reset effects (only)
reset Reset colors and effects (restore defaults)
underline Start underline text effect
=============== =================================================

The following color names are supported:

========= ============================================
Name Description
========= ============================================
black
blue
cyan
default Default color for foreground or background
green
magenta
red
white
yellow
========= ============================================
========= ============================================
Name Description
========= ============================================
black
blue
cyan
default Default color for foreground or background
green
magenta
red
white
yellow
========= ============================================

When using colors, the developer should remember that users will
change the foreground and background colors of terminal session


+ 1
- 0
contrib/libxo/doc/index.rst View File

@@ -38,6 +38,7 @@ libxo ships as part of FreeBSD.
field-modifiers
field-formatting
api
encoders
xo
xolint
xohtml


+ 0
- 27436
contrib/libxo/doc/libxo-manual.html
File diff suppressed because it is too large
View File


+ 0
- 3990
contrib/libxo/doc/libxo.txt
File diff suppressed because it is too large
View File


+ 51
- 51
contrib/libxo/doc/options.rst View File

@@ -33,36 +33,36 @@ Option Keywords
Options is a comma-separated list of tokens that correspond to output
styles, flags, or features:

=============== =======================================================
Token Action
=============== =======================================================
color Enable colors/effects for display styles (TEXT, HTML)
colors=xxxx Adjust color output values
dtrt Enable "Do The Right Thing" mode
flush Flush after every libxo function call
flush-line Flush after every line (line-buffered)
html Emit HTML output
indent=xx Set the indentation level
info Add info attributes (HTML)
json Emit JSON output
keys Emit the key attribute for keys (XML)
log-gettext Log (via stderr) each gettext(3) string lookup
log-syslog Log (via stderr) each syslog message (via xo_syslog)
no-humanize Ignore the {h:} modifier (TEXT, HTML)
no-locale Do not initialize the locale setting
no-retain Prevent retaining formatting information
no-top Do not emit a top set of braces (JSON)
not-first Pretend the 1st output item was not 1st (JSON)
pretty Emit pretty-printed output
retain Force retaining formatting information
text Emit TEXT output
underscores Replace XML-friendly "-"s with JSON friendly "_"s
units Add the 'units' (XML) or 'data-units (HTML) attribute
warn Emit warnings when libxo detects bad calls
warn-xml Emit warnings in XML
xml Emit XML output
xpath Add XPath expressions (HTML)
=============== =======================================================
=============== =======================================================
Token Action
=============== =======================================================
color Enable colors/effects for display styles (TEXT, HTML)
colors=xxxx Adjust color output values
dtrt Enable "Do The Right Thing" mode
flush Flush after every libxo function call
flush-line Flush after every line (line-buffered)
html Emit HTML output
indent=xx Set the indentation level
info Add info attributes (HTML)
json Emit JSON output
keys Emit the key attribute for keys (XML)
log-gettext Log (via stderr) each gettext(3) string lookup
log-syslog Log (via stderr) each syslog message (via xo_syslog)
no-humanize Ignore the {h:} modifier (TEXT, HTML)
no-locale Do not initialize the locale setting
no-retain Prevent retaining formatting information
no-top Do not emit a top set of braces (JSON)
not-first Pretend the 1st output item was not 1st (JSON)
pretty Emit pretty-printed output
retain Force retaining formatting information
text Emit TEXT output
underscores Replace XML-friendly "-"s with JSON friendly "_"s
units Add the 'units' (XML) or 'data-units (HTML) attribute
warn Emit warnings when libxo detects bad calls
warn-xml Emit warnings in XML
xml Emit XML output
xpath Add XPath expressions (HTML)
=============== =======================================================

Most of these option are simple and direct, but some require
additional details:
@@ -94,25 +94,25 @@ Brief Options
The brief options are simple single-letter aliases to the normal
keywords, as detailed below:

======== =============================================
Option Action
======== =============================================
c Enable color/effects for TEXT/HTML
F Force line-buffered flushing
H Enable HTML output (XO_STYLE_HTML)
I Enable info output (XOF_INFO)
i<num> Indent by <number>
J Enable JSON output (XO_STYLE_JSON)
k Add keys to XPATH expressions in HTML
n Disable humanization (TEXT, HTML)
P Enable pretty-printed output (XOF_PRETTY)
T Enable text output (XO_STYLE_TEXT)
U Add units to HTML output
u Change "-"s to "_"s in element names (JSON)
W Enable warnings (XOF_WARN)
X Enable XML output (XO_STYLE_XML)
x Enable XPath data (XOF_XPATH)
======== =============================================
======== =============================================
Option Action
======== =============================================
c Enable color/effects for TEXT/HTML
F Force line-buffered flushing
H Enable HTML output (XO_STYLE_HTML)
I Enable info output (XOF_INFO)
i<num> Indent by <number>
J Enable JSON output (XO_STYLE_JSON)
k Add keys to XPATH expressions in HTML
n Disable humanization (TEXT, HTML)
P Enable pretty-printed output (XOF_PRETTY)
T Enable text output (XO_STYLE_TEXT)
U Add units to HTML output
u Change "-"s to "_"s in element names (JSON)
W Enable warnings (XOF_WARN)
X Enable XML output (XO_STYLE_XML)
x Enable XPath data (XOF_XPATH)
======== =============================================

.. index:: Colors

@@ -145,7 +145,7 @@ For example consider the following xo_emit call::
xo_emit("{C:fg-red,bg-green}Merry XMas!!{C:}\n");

To turn all colored output to red-on-blue, use eight pairs of
"red/blue" mappings separated by "+"s::
"red/blue" mappings separated by plus signs ("+")::

--libxo colors=red/blue+red/blue+red/blue+red/blue+\
red/blue+red/blue+red/blue+red/blue
@@ -159,6 +159,6 @@ to green (the third mapping)::
Consider the common situation where blue output looks unreadable on a
terminal session with a black background. To turn both "blue"
foreground and background output to "yellow", give only the fifth
mapping, skipping the first four mappings with bare "+"s::
mapping, skipping the first four mappings with bare plus signs ("+")::

--libxo colors=++++yellow/yellow

+ 9
- 0
contrib/libxo/doc/top-link.html.in View File

@@ -0,0 +1,9 @@
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Refresh" content="0; url=@LIBXO_VERSION@/html/index.html" />
</head>
<body>
<p>The current libxo version is <a href="@LIBXO_VERSION@/html/index.html">@LIBXO_VERSION@</a>.</p>
</body>
</html>

+ 32
- 9
contrib/libxo/doc/xo.rst View File

@@ -1,5 +1,5 @@

.. index:: --libxo, xo
.. _xo:

The "xo" Utility
================
@@ -12,9 +12,7 @@ The style of output can be selected using a specific option: "-X" for
XML, "-J" for JSON, "-H" for HTML, or "-T" for TEXT, which is the
default. The "--style <style>" option can also be used. The standard
set of "--libxo" options are available (see :ref:`options`), as well
as the `LIBXO_OPTIONS`_ environment variable.

.. _`LIBXO_OPTIONS`: :ref:`libxo-options`
as the :ref:`LIBXO_OPTIONS <libxo-options>` environment variable.

The `xo` utility accepts a format string suitable for `xo_emit` and
a set of zero or more arguments used to supply data for that string::
@@ -23,12 +21,15 @@ a set of zero or more arguments used to supply data for that string::

TEXT:
The fish weighs 6 pounds.

XML:
<name>fish</name>
<weight>6</weight>

JSON:
"name": "fish",
"weight": 6

HTML:
<div class="line">
<div class="text">The </div>
@@ -54,6 +55,7 @@ by the '/' character::
</b>
</a>
</top>

JSON:
"top": {
"a": {
@@ -72,17 +74,19 @@ then close tags. The `--depth` option may be used to set the
depth for indentation. The `--leading-xpath` may be used to
prepend data to the XPath values used for HTML output style::

EXAMPLE;
EXAMPLE:
#!/bin/sh
xo --open top/data
xo --depth 2 '{:tag}' value
xo --close top/data

XML:
<top>
<data>
<tag>value</tag>
</data>
</top>

JSON:
"top": {
"data": {
@@ -102,14 +106,18 @@ Use the `--top-wrap` option to ensure any top-level object details are
handled correctly, e.g. wrap the entire output in a top-level set of
braces for JSON output.

EXAMPLE;
::

EXAMPLE:
#!/bin/sh
xo --top-wrap --open top/data
xo --depth 2 'First {:tag} ' value1
xo --depth 2 --continuation 'and then {:tag}\n' value2
xo --top-wrap --close top/data

TEXT:
First value1 and then value2

HTML:
<div class="line">
<div class="text">First </div>
@@ -118,6 +126,7 @@ braces for JSON output.
<div class="text">and then </div>
<div class="data" data-tag="tag">value2</div>
</div>

XML:
<top>
<data>
@@ -125,6 +134,7 @@ braces for JSON output.
<tag>value2</tag>
</data>
</top>

JSON:
{
"top": {
@@ -150,7 +160,7 @@ them. Each of these options take a `name` parameter, providing the
name of the list and instance.

In the following example, a list named "machine" is created with three
instances:
instances::

opts="--json"