Change the default Unicode error errno.

In times long past, EINVAL was chosen for the errno returned when a
Unicode problem was detected.  This overloads EINVAL, particularly when
using POSIX system calls.  Unfortunately, the overloading makes it very
difficult to recover from some errors as there is no way to know the
difference between a "real" EINVAL and a Unicode problem EINVAL.

Over 10 years of data shows that we *RARELY* encounter Unicode errors.
Inspecting the source base shows the code segments that do check for
errno after an error either:

1) Post the errno value and die.
2) Have switch handle cases but also have a default catcher.
3) Have an if/else waterfall with a final else.

Another observation is that Unicode routines are not involved with
anything that would return a math related errno (e.g. ERANGE, EDOM).

Changing the value of UNICODE_CONVERSION_ERRNO to ERANGE and avoid
any overloading.

diff --git a/open-vm-tools/lib/include/unicodeBase.h b/open-vm-tools/lib/include/unicodeBase.h
index 2dbea50bc3fe9759ebb9454512abedbe5c25c77b..2034905a7f235d053250144d8a11fc0926d280db 100644 (file)
--- a/open-vm-tools/lib/include/unicodeBase.h
+++ b/open-vm-tools/lib/include/unicodeBase.h
@@ -1,5 +1,5 @@
 /*********************************************************
- * Copyright (C) 2007-2019 VMware, Inc. All rights reserved.
+ * Copyright (C) 2007-2020 VMware, Inc. All rights reserved.
  *
  * This program is free software; you can redistribute it and/or modify it
  * under the terms of the GNU Lesser General Public License as published
@@ -62,12 +62,14 @@ extern "C" {
 
 
 /*
- * In contexts where an errno makes sense, use this
- * to report conversion failure.
+ * In contexts where an errno makes sense, use this to report conversion
+ * failure. ERANGE is chosen since no Unicode routines deal with math; this
+ * avoids overloading an errno that may a specific meaning for a certain
+ * routine.
  */
 
 #ifndef _WIN32
-#define UNICODE_CONVERSION_ERRNO EINVAL 
+#define UNICODE_CONVERSION_ERRNO ERANGE
 #endif