HGFS: Partial fix for corruption when using different file handles to the same file
File is corrupted while our customer using two threads to read or write a
file via Linux HGFS client. This is because the read handle interferes with
the writes by causing a revalidation of the inode's file attributes. These
were mishandled wrt to flushing out the cached pages. If the new attributes
were different for modification time or file size then the pages in the cache
were invalidated. This causes pages of valid data to be thrown away and the
writes lost. Therefore resulting in a file with gaps of blocks of zero bytes
where writes were not sent to the HGFS server.
This is fixed by replicating what NFS does in this regard, which is only invalidate
the cache if the HGFS server returned file size only differs from the cached
inode value and only then if the new size is greater.
Cleaned up the write begin and end which was initially causing problems due
to very buggy code. Have now based this on simplicity from fs/libfs.c and
the simple_write_begin/simple_write_end which shows what the minimal settings
should do handling writes to pages and partial page writes.
These can be viewed under you favorite linux source cross-reference website.
Fix AssertOnCompileFailed for GCC 4.8 (kernel 3.10+)
GCC 4.8 now includes -Wunused-local-typedefs as part of -Wall. This
causes it to spew a gazillion warnings when compiling our modules on
3.10+ kernels. Because it's a typedef we can't use the (void) cast
trick, so we need to use attributes. This change introduces
UNUSED_TYPE to vm_basic_types.h and uses it where appropriate to make
the compiler happy when building our modules.
Right now we use vmalloc()/alloc_page() for our guest queuepair pages
(and bitmap page) and then do a virt_to_phys() before passing them down
to the device. That's not going to work if DMA remapping is enabled,
since the IOMMU has no idea about the mappings. Switch to
pci_alloc_consistent() instead. We still allocate each page
individually, since there's no guarantee that we'll get a contiguous
block of physical for an entire queuepair (especially since we allow up
to 128 MiB!).
Also made split between guest and host in the kernelIf struct much
clearer. Now it's obvious which fields are which.
Finally, drop the PINNED flag from Linux. That was only ever used by
vVol-Filter, and that's been moved over vPageChannel, which doesn't use
queuepairs but has custom rings.
Once this is in I'll port it to upstream. I considered doing upstream
only, but that would mean pre-3.9 guests would be unable to use
IOMMU support. We should really freeze the in-house driver after
this...
Enhancing existing log statements to be useful, moving open flags utility function
into the utility file as in the future it should be used by open and create
entry points.
VMI was gone a long time ago. There are a few remnants
still present.
- includeCheck has a VMIROM macro that can go, plus all
locations it was copy-n-pasted into
- vmcore/public/paravirtualInterface.h was totally unreferenced.
The Linux HGFS client miscalculated the number of 512 byte blocks to set
in the inode for the file size. It was incorrectly using the HGFS
hardcoded block size instead of 512, which was 1024. This causes the
block count to be half the correct number and so du on a file reports
the incorrect size. fstat also retrieves this block count in the stat
structure so it is likely more applications will break.
Change GuestLIB SDK to use new RpcChannel API.
1) Change RpcOut_* calls to RpcChanneel_* calls.
2) guestlib SDK now has extra dependency on GLIB and vmtoolslib as
secure guestRPC brings a lot new dependencies (rpcin, asyncsocket, ssl,
datamap, hashmap etc).
3) ship new glib and vmtoolslib for linux and windows packages.
4) merge logging functions (Debug(), Warning()) in vmtools and guestlib
into one to avoid having same function names defined in two libs.
5) update build in open vm tools.
6) app monitor needs to do similar thing, which will be handled by HA
team.
The kernel buffer holding the symlink name was being freed incorrectly
when it was used by the VFS layer. This resulted in corruption and
invalid names being used when trying to lookup the symlink's target.
The HgfsFollowlink should not be calling vfs_follow_link but calling
nd_set_link to save the link target name that HGFS allocated. To deal
with the release of the name, HgfsPutlink has now been added which the
VFS layer will call as needed and this function retrieves the name from
the name structure using nd_get_link and releases the buffer. Then for
completeness calls nd_set_link with NULL to clear it from the name
structure. The VFS layer internally calls vfs_follow_link after the
HgfsFollowlink call passing the link target name from the name object
that HGFS stored. Hence, why HGFS should not call it directly.
When doing partial page writes the Linux HGFS client would trash the
initial part of the page leading up to the start of the write and also
trash the remaining part of the page after the written segment of the
page.
This trashing of the initial part of the page was incorrect and
overwrites earlier partial page writes to that page. This was also
incorrect as the arguments for the offsets into the page were
miscalculated.
. Fix FileIO_CloseAndUnlink() to handle unlink failures.
. Add Debuan 7,1 to the list of known OSes
. changes in shared code that don't affect open-vm-tools functionality
Fix parsing logic in guestInfo to parse vlan configured NICs
The regular expression used to parse routes /proc/net/route and
/proc/net/ipv6_route fail when the device names contain "." eg: eth0.80
which is how the interface names appear when we have VLANs configured.
This fix modifies the regex to handle this case by parsing until a
whitespace character is encountered.
Use g_debug instead of g_warning in RpcChannel_Error
With esx5.1GA Tools, when there are more two Terminal Sessions
connect to a Windows VM, each TS sould have one vmtoolsd running
but actually only max two vmtoolsd run. The Windowds Application
Event log fills up with the warning messages of
"[vmusr:vmusr] Error in the RPC receive loop: RpcIn: Unable to send."
repeatly until close some TS' to have max two TS connected. Some
customers had vmtoolsd crashed but some didn't. This message is from
RpcChannel_Error. After replace g_warning() by g_debug(), the problem
is resolved. This is a workaround for esx5.1x.
1) Fix some assertion for reconnect.
2) Call VMCISock_GetAFValueFd so we do not hold vsock usage all the time.
2) Cut off the dependency of SIGUSR1 with dndcp plugin.
3) Change function name vmware_unblock_vmware_user to send_vmware_user_sigusr1.
Avoid logging non-fatal messages without logging system initialization.
Logs that come before logging system intialization spam the
console output. Therefore, we make the logging a no-op for
non-fatal messages until logging system has been initialized.
Usually, we initialize the logging system when process starts up,
so we will swallow only those non-fatal log messages that are
generated during very early stages of process initialization.
While cleaning up the usage of the HGFS packet abstraction it meant that
the arguments to the HGFS server packet send callback could be cleaned up.
The arguments for the packet buffer and size are now unused as this too
arguments are extracted from the HGFS packet object.
Therefore, remove the unused arguments from the send callback.
. Tools,DND,X11: Constrain detection window to desktop work area
. dndcpx11: Adjust style to match Hosted UI guidelines
. changes in shared code that don't affect open-vm-tools functionality
Deal with the broken reply packet part of the abstraction.
Fix the get reply packet function call to separate out the reply data
size passed in and the reply packet buffer size which is returned.
Add the reply data size field in the HgfsPacket object so that the reply
data size and reply buffer size can be separated out and not overloaded.
This allows the transport channel send routines to extract the reply
details from the packet object and not need the buffer and size
arguments. The send function channel callback API will be cleaned up in
a separate change.
The get reply function for the packet now sets the meta packet data size
too when it is used for the reply packet. This is used by the VMCI
channel which now only maps and copies the correct amount of reply data
into the guest VM shared memory.
The reply packet size is randomly overwritten from the total packet size
available for the reply to hold the amount of written data in the reply.
This however, does not have any effect, since, the reply data size is
passed to the send function which then knows how much data to send, and
for VMCI, the meta part of the packet is used to transfer data to the
guest shared memory so the reply part is essentially unused. This is now
deleted.
This change is supporting a 64bit MPN for hosted products.
Assumptions taken:
1. Kindly note that currently we do not have systems supporting 64bit
memory. So hosted vmmon 64 bit MPN is a container to store whatever
monitor passes to it. The underlying assumption is that the higher
32bits will be zero.
2. Physmem tracker still does page walk using 32 bit values and we have
put checks to make sure that MPN64 is just a container with actually 32
bit values. This was a design decision taken after talking to kevinc and
jpool.
3. The CrossPage code is still 32 bit and I typecast MPN64 to accept 32
bit values for the crosspage code.
4. Vmkernel already has another definition of MPN64 and I have retained
their definition for vmkernel specific code.
typedef uint8 * MPN64;
typedef uint64 MPN64;
5. Additional changes were required in bora-vmsoft vmballoon module
which earlier used MPN and called functions expecting MPN64 as an input.
I have however *NOT* changed the PageHandle code of Vmballoon. I have
typecasted to MPN64 where common MacOS page handling functions which
require MPN64 are used.
Pass RPCI channel info to guestrpc command hanlders.
Since now we have secure guestRpc, some commands like namespace manager
needs to handle vsocket/backdoor connections differently. main changes
are in guest_rpc.c, guest_rpc.h, and namespaceMgr.c. Changes in other
files just function sinature changes.
Tools/DnD X11: Logging improvements and cosmetic touchups.
===== Logging =====
A large chunk of the Tools DND code included function call tracing via
log entries at the debug level. However, it had some problems, such as
generating a lot of noise in all buildtypes. (Remember that Tools only
has 3-4 log levels: debug, info, warning, and error.) There was also a
problem with inconsistency w/r/t logging when a function went out of
scope.
To address this, I whipped up a dummy tracing object and shove it in
bora-vmsoft/lib/public. By including tracer.hh, one just inserts
TRACE_CALL(); at the beginning of a function to log entry/exit.
Additionally, it works only in developer builds; it's a ifdef'd out
everywhere else.
Another thing I did was centralize logging of GuestCopyPasteMgr's
session ID and state updates. We now long old/new values upon update.
More of the DND code has been fixed to log to the dndcp GLib log domain,
making it easier for folks to isolate dndcp logs. (This explains the
switch from Debug to g_debug.)
===== Readability ====
There was a list of some 10+ sigc signals that, due to indentation and
such, was kinda hard to read & easily skim. I replaced this with a
macro which reduces each signal hookup to one line.
Switch utf::string from g_utf8_validate to Unicode_IsBufferValid.
In the version of glib we currently use g_utf8_validate rejects strings
containing the non-characters U+FFFE and U+FFFF. This was marked as a
bug and resolved (see https://bugzilla.gnome.org/show_bug.cgi?id=694669)
but until that is released we need a workaround. Our internal UTF-8
validation routine accepts these characters as valid so use that
instead.
Fix VMHSUtilFileMove and clarify comments of File_Replace
VMHSUtilFileMove has been broken since 2008 and no one has noticed,
so I would be happy to remove the entire mv command from VMHS if
no one would miss it.
Change the lspci parameters to avoid crash dangerous and offer more readable information.
Currently our WS `vm-support` runs:
lspci -H1 -M
lspci -H1 -M -vn
The -H parameter is dangerous. In better case it will crash your server,
in worse case it will corrupt your data.
So change them to:
runcmd "lspci -M -vvv -nn -xxxx" "/tmp/lspci1.$$.txt"
runcmd "lspci -t -v -nn -F /tmp/lspci1.$$.txt" "/tmp/lspci2.$$.txt"
runcmd "lspci -vvv -nn" "/tmp/lspci3.$$.txt"
Adding in the new fields to indicate the size of the valid data, not the
size of the buffer. Previously, the two were combined which meant that
the buffer size was ultimately used instead making the data copies very
inefficient.
Another transitional change to get this stuff moved over to handle the
total pack size and valid data sizes separately. Currently, VMCI has to
use the total packet sizes only and so maps in 16 pages and copies all
the data back and forth because it can't tell how much is actually
valid. When I switch the VMCI transport header to support V2 it will
have the correct data size and packet ID - asynchronous will then work
too.
Mac OS Tools: Make system-wide services launchd-compliant
Since day 1, the way we have started the tools services has been: at
boot, launchd executed the process "services.sh --start", which:
1) Loaded kexts.
2) Started the long-lived vmsvc as a daemon.
3) Exited.
According to http://developer.apple.com/library/mac/#documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html ,
launchd sends it a SIGTERM. See "SIGTERM".
This change fixes the above two issues, by turning things upside down:
since vmsvc is already a long-lived process, let's:
o Have launchd start vmsvc directly, without daemonizing it, just like
launchd already does with the vmusr process.
o Modify vmsvc so that the first thing it does is load the kexts.
I kept the "services.sh --stop" interface (I just re-implemented it
differently), because it is useful to testers, but more importantly
because it is called from two places:
The tools preinstall package script (possibly in a tools package older
than this change).
The tools uninstallation script.
This change makes use of __thread on Linux for storing the vthread
base pointer and the thread ID. We still also store the value with
pthreads_setspecific so that we have consistent behavior on thread
destruction (using the pthreads destructor). But this allows the
VThread_CurID path to skip loading the pthread key, skip checking if
the key is initialized, and skip the call into pthread_getspecific.
(With VThread_CurID taking its own fast path, it's not clear there's
much benefit at all from optimizing the path for the base pointer, but
I did so for consistency's sake.)
I tried using my USB bandwidth test to measure the performance impact
of this on a Linux host. Unfortunately I found the test to be much
more noisy under Linux. The results claimed 1.6% faster overall, but
I have almost no faith in that number.
Still, having inspected the new code and looked at the contents of
pthread_getspecific, I think it's safe to claim that this should be
better overall even if its difficult to quantify how much. And though
it makes the code slightly more crufty, such ugliness is confined to
two functions.
Sometimes when we make preprocessor tables we have to blow past 80
columns. But in x86cpuid.h we had a table with plenty of internal
whitespace which was 81 columns long. No reason to have to suffer
with that.
The HGFS server sets the packet's ID field as the HGFS protocol request
ID. This should not occur and it overloads that the packet ID which
should be private to the transport channel which has nothing to do with
the HGFS request ID. The HGFS request ID has it's own field in the HGFS
header used by the request and reply.
Remove the HGFS server modifying the packet ID. This is currently safe,
as the clients do not receive any asynchronous replies from any HGFS
server where the packet ID in the reply is used.
Had a few spare cycles, so tried compiling Tools with gcc-4.4.3.
This change fixes all compilation errors. (It does NOT do the
compiler upgrade, hence the minimal testing.)
All changes are mechanical / should be correct at compile-time.
appUtilFileTypes.h: hash_map was never a standard, use the
standardized (std::tr1) unordered_map. Which removes ifdefs too.
most other files: character strings are "const" in C++.
VM2VM VMCI made a reappearance in 5.5 because vHadoop was planning to
use it, but it was eventually rejected. We left the code in so as not
to disturb stability, and simply disabled it in the UI. But let's
make doubly sure by flipping the kill-switch, so that our customers
don't accidentally enable it and then complain about it being removed
from 5.1, coming back in 5.5 and then disappearing again.
This cleans up some of the inflexibility and inconsistencies in the use
of the mappings of the packet buffers for the meta data (Hgfs header and
commmand arguments) and the data component.
This splits out hanlding the iov mappings into two smaller utility
functions (map and unmap) since the code is replicated in multiple
places.
The copy from and too an allocated buffer into and from an iov array is
split out from the map and unmap functionality and also a second routine
is created to copy from the iov array to a buffer which was previously
missing.
To achieve this I have also added a mapped iov count for the meta and
data iov components of the HgfsPacket object. This helps track the state
of when mappings are available and not. Previously, this was coded by
assumption of what the GetBuf call did.
In follow up changes the data will have a total buffer size and a data
size for each of the meta and data components. This will allow for
optimal movement of data between buffer and iov array.
Avoid a memory dereference when getting the current thread ID.
As part of some recent benchmarking and ensuing discussions a couple
of overheads in VThread_CurID have been identified.
(1) VThread_CurID ultimately gets a pointer to a per thread
structure that contains various bits of state including the
thread ID. After getting that pointer it must read the thread
ID out of memory. This could be avoided if the thread local
value we stored was the thread ID itself.
(2) Before calling into the host's API for thread local storage,
vthreadBase.c must get the host key, check if it is initialized
and initialize it if necessary. Then once it gets the thread
local value from the host API it must check if that has been
initialized. These checks could be avoided if we forced clients
to initialize the thread before calling functions like
VThread_CurID.
(3) On some (all?) Linux pthread implementations pthread_getspecific
itself can be heavyweight. Both Windows and Linux offer
alternatives to make thread local storage cheaper. (OS X on the
other hand provides a very fast implementation of
pthread_getspecific: pretty much one instruction plus the
function call overhead.)
The first two overheads came up in my profiling of USB workloads on OS
X while Kevin raised the third issue in the following discussions.
This change attempts to eliminate the first overhead, but to do so in
a way that helps set up the code for attacking the remaining two. In
particular it introduces a second thread local variable to store the
thread ID. For the time I left the thread ID in the other structure
as well and verify that they stay in sync. We could get rid of it,
but it's low cost and I suspect (though I have no proof) that it could
be useful in debugging.
As part of this I've rearranged some of the initialization code both
for the TLS keys as well as for initializing the TLS data. This is
useful because I wanted both pieces of state to get set together (and
mutated together -- yes our threads change IDs during their life).
And, with respect to the pthread keys, it's important to make sure the
base key gets allocated before the thread ID key. This ensures it
gets destroyed first which allows us to keep an ASSERT to make sure
that the two thread IDs stay in sync.
This does run into one wrinkle because of the lazy thread
initialization. The default value for uninitialized state is NULL
which, inconveniently, is a valid vthread ID. So instead of actually
storing the thread ID, we store the thread ID incremented by one.
This means the default/uninitialized value will show up as -1. We can
avoid this after addressing (2) from above, but in the meantime
trading a memory read for an ALU operation is still a nice win.
Disk shrink is a long running operation with progress reporting. The
"progress" string is visible to users and therefore needs to be
internationalized.
The transport session is only ever used in the common HGFS server code.
The scope should be constrained to that file and beyond that treated as
an opaque type.
Make some of the transport session functions static to the common HGFS
server file and remove the declarations from the common header file.
Clean up the packet utility routines to only pass the arguments that are
required. This means we only need to use the HGFS server channel
callbacks and not the whole HGFS server transport session object.
The kernel changes broke the asynchronous read and write HGFS
client code which picks out the dentry from within the kiocb struct.
Include the header file directly now for newer kernel versions.
In particular we see const void * and void const * intermixed and so
now make the server code consisten in its use and stick with the
const void * format.
Complete some tidy up of the input params created from the HgfsPacket.
This can be contained to only the HgfsServer code and not required to be
exposed beyond that. So this removes the one usage from the parameter
pack and unpack code which was in the unpack write request.
As I was modifying up the HGFS server write request I correct a couple of
const char * to const void * for the write data.
I removed a HSPU_PutPacket declaration as the function did not exist.
HGFS: Fix server check for minimum sizes of session requests
The HGFS server check for the minimum protocol request sizes for create
and destroy session was incorrect. The full request including the reserved
fields must be sent by the clients. Currently they all do that, including any
shipping versions.
Now the HGFS server packet abstraction has moved from an invalid
const char * pointer for the input parameters set from the HgfsPacket
abstraction we make the pack functions correct too.
This fixes the packet header which can be of two different types to be
a const void * now and not the incorrect const char *.
HGFS: Clean up HGFS server packet abstraction part II
Clean up the input params object which reuses the packet abstraction
field names which are generic because they are opaque outside of the HGFS
server. Since the input params are wholly contained within the HGFS server
and have specific meaning in the HGFS server context, we give the fields
the HGFS meaningful names.
The HGFS packet object passed between the transport channel and the HGFS
server is in need of some clean up so that it can be made much more efficient.
This just closes out some unnecessary public functions and makes them static.
HGFS: Fix the server to close sessions being invalidated
When the HGFS server running in the tools is left with any sessions
open due to the clients not terminating them cleanly they will be marked
inactive. The HGFS server callback to invalidate them will periodically
be called and after a brief period will terminate those sessions.
These sessions are not closed first, thus triggering the assert in the
session exit function which checks the session's state.
Fix is to close the inactive sessions being terminated in the HGFS
server invalidate callback.
HGFS: Fix Linux client to work with Perforce versioning
The problem is that perforce client uses rename operation upon a temp
file which has read-only attributes. The operation then fails with
permission denied.
The fix removes the read only attribute and retries the rename operation
again as per the delete operation. This occurs on Windows HGFS servers
as the target of the rename operation to be replaced has the read only
attribute set. The prevents a rename even if they want to overwrite the
target if it exists. So removing the read only flag is required.
. Scrub sensitive data in VIX before freeing it
. Fix memory leak in VMCISock_GetAFValueFd()
. changes in shared code that don't affect open-vm-tools functionality
Use VMCI/Vsocket instead of backdoor for GuestRPC RPCI channels. For
privileged channels, the guest side can bind port to less than 1024, VMX
can then verify the binding and enforce privileged commands can only be
ran by privileged users.
projects / thirdparty / open-vm-tools.git / log
