Had a few spare cycles, so tried compiling Tools with gcc-4.4.3.
This change fixes all compilation errors. (It does NOT do the
compiler upgrade, hence the minimal testing.)
All changes are mechanical / should be correct at compile-time.
appUtilFileTypes.h: hash_map was never a standard, use the
standardized (std::tr1) unordered_map. Which removes ifdefs too.
most other files: character strings are "const" in C++.
VM2VM VMCI made a reappearance in 5.5 because vHadoop was planning to
use it, but it was eventually rejected. We left the code in so as not
to disturb stability, and simply disabled it in the UI. But let's
make doubly sure by flipping the kill-switch, so that our customers
don't accidentally enable it and then complain about it being removed
from 5.1, coming back in 5.5 and then disappearing again.
This cleans up some of the inflexibility and inconsistencies in the use
of the mappings of the packet buffers for the meta data (Hgfs header and
commmand arguments) and the data component.
This splits out hanlding the iov mappings into two smaller utility
functions (map and unmap) since the code is replicated in multiple
places.
The copy from and too an allocated buffer into and from an iov array is
split out from the map and unmap functionality and also a second routine
is created to copy from the iov array to a buffer which was previously
missing.
To achieve this I have also added a mapped iov count for the meta and
data iov components of the HgfsPacket object. This helps track the state
of when mappings are available and not. Previously, this was coded by
assumption of what the GetBuf call did.
In follow up changes the data will have a total buffer size and a data
size for each of the meta and data components. This will allow for
optimal movement of data between buffer and iov array.
Avoid a memory dereference when getting the current thread ID.
As part of some recent benchmarking and ensuing discussions a couple
of overheads in VThread_CurID have been identified.
(1) VThread_CurID ultimately gets a pointer to a per thread
structure that contains various bits of state including the
thread ID. After getting that pointer it must read the thread
ID out of memory. This could be avoided if the thread local
value we stored was the thread ID itself.
(2) Before calling into the host's API for thread local storage,
vthreadBase.c must get the host key, check if it is initialized
and initialize it if necessary. Then once it gets the thread
local value from the host API it must check if that has been
initialized. These checks could be avoided if we forced clients
to initialize the thread before calling functions like
VThread_CurID.
(3) On some (all?) Linux pthread implementations pthread_getspecific
itself can be heavyweight. Both Windows and Linux offer
alternatives to make thread local storage cheaper. (OS X on the
other hand provides a very fast implementation of
pthread_getspecific: pretty much one instruction plus the
function call overhead.)
The first two overheads came up in my profiling of USB workloads on OS
X while Kevin raised the third issue in the following discussions.
This change attempts to eliminate the first overhead, but to do so in
a way that helps set up the code for attacking the remaining two. In
particular it introduces a second thread local variable to store the
thread ID. For the time I left the thread ID in the other structure
as well and verify that they stay in sync. We could get rid of it,
but it's low cost and I suspect (though I have no proof) that it could
be useful in debugging.
As part of this I've rearranged some of the initialization code both
for the TLS keys as well as for initializing the TLS data. This is
useful because I wanted both pieces of state to get set together (and
mutated together -- yes our threads change IDs during their life).
And, with respect to the pthread keys, it's important to make sure the
base key gets allocated before the thread ID key. This ensures it
gets destroyed first which allows us to keep an ASSERT to make sure
that the two thread IDs stay in sync.
This does run into one wrinkle because of the lazy thread
initialization. The default value for uninitialized state is NULL
which, inconveniently, is a valid vthread ID. So instead of actually
storing the thread ID, we store the thread ID incremented by one.
This means the default/uninitialized value will show up as -1. We can
avoid this after addressing (2) from above, but in the meantime
trading a memory read for an ALU operation is still a nice win.
Disk shrink is a long running operation with progress reporting. The
"progress" string is visible to users and therefore needs to be
internationalized.
The transport session is only ever used in the common HGFS server code.
The scope should be constrained to that file and beyond that treated as
an opaque type.
Make some of the transport session functions static to the common HGFS
server file and remove the declarations from the common header file.
Clean up the packet utility routines to only pass the arguments that are
required. This means we only need to use the HGFS server channel
callbacks and not the whole HGFS server transport session object.
The kernel changes broke the asynchronous read and write HGFS
client code which picks out the dentry from within the kiocb struct.
Include the header file directly now for newer kernel versions.
In particular we see const void * and void const * intermixed and so
now make the server code consisten in its use and stick with the
const void * format.
Complete some tidy up of the input params created from the HgfsPacket.
This can be contained to only the HgfsServer code and not required to be
exposed beyond that. So this removes the one usage from the parameter
pack and unpack code which was in the unpack write request.
As I was modifying up the HGFS server write request I correct a couple of
const char * to const void * for the write data.
I removed a HSPU_PutPacket declaration as the function did not exist.
HGFS: Fix server check for minimum sizes of session requests
The HGFS server check for the minimum protocol request sizes for create
and destroy session was incorrect. The full request including the reserved
fields must be sent by the clients. Currently they all do that, including any
shipping versions.
Now the HGFS server packet abstraction has moved from an invalid
const char * pointer for the input parameters set from the HgfsPacket
abstraction we make the pack functions correct too.
This fixes the packet header which can be of two different types to be
a const void * now and not the incorrect const char *.
HGFS: Clean up HGFS server packet abstraction part II
Clean up the input params object which reuses the packet abstraction
field names which are generic because they are opaque outside of the HGFS
server. Since the input params are wholly contained within the HGFS server
and have specific meaning in the HGFS server context, we give the fields
the HGFS meaningful names.
The HGFS packet object passed between the transport channel and the HGFS
server is in need of some clean up so that it can be made much more efficient.
This just closes out some unnecessary public functions and makes them static.
HGFS: Fix the server to close sessions being invalidated
When the HGFS server running in the tools is left with any sessions
open due to the clients not terminating them cleanly they will be marked
inactive. The HGFS server callback to invalidate them will periodically
be called and after a brief period will terminate those sessions.
These sessions are not closed first, thus triggering the assert in the
session exit function which checks the session's state.
Fix is to close the inactive sessions being terminated in the HGFS
server invalidate callback.
HGFS: Fix Linux client to work with Perforce versioning
The problem is that perforce client uses rename operation upon a temp
file which has read-only attributes. The operation then fails with
permission denied.
The fix removes the read only attribute and retries the rename operation
again as per the delete operation. This occurs on Windows HGFS servers
as the target of the rename operation to be replaced has the read only
attribute set. The prevents a rename even if they want to overwrite the
target if it exists. So removing the read only flag is required.
. Scrub sensitive data in VIX before freeing it
. Fix memory leak in VMCISock_GetAFValueFd()
. changes in shared code that don't affect open-vm-tools functionality
Use VMCI/Vsocket instead of backdoor for GuestRPC RPCI channels. For
privileged channels, the guest side can bind port to less than 1024, VMX
can then verify the binding and enforce privileged commands can only be
ran by privileged users.
HGFS: Clean up the VMCI transport request and reply headers part I
This is the first part of cleaning up the HGFS VMCI transport request and
reply headers. This deals with the reply header only which is almost identical
to the client request VMCI header but not quite - frustratingly.
It really is not ideal to use a reply header that is different from
the request header and has fields that require corresponding information from
the request header which are missing. Consequently, the request header is
going to be ramped up to a version 2 which includes all the information for both
request and reply and be extensible.
The MFENCE instruction ensures that all loads and stores that
preceed it are "globally visible" before any load or store that
follow it. However, we don't have a compiler barrier in
Atomic_MFence and so the compiler can move code and defeat this.
While this bug has probably been around for quite a while, it
only came up recently now that the AtomicEpilogue() is compiled
out. That contains a compiler memory barrier, and while it's
not technically sufficient (need barrier before the MFENCE too),
it was apparently enough to discourage the code motion in
practice.
Previously, when a transport session was disconnected any notifications
for the HGFS session were then removed if notification module was enabled.
However, this is grossly inadequate, as an HGFS session can come and go driven by
a client's protocol requests. The notifications are per HGFS session which is
completely independent from the transport session. The only reason the transport
session comes into play is whether it has the transport characteristics to support
the bidirectional nature of the notification feature.
When an HGFS session is destroyed whether it is from a tranport connection
session disconnect or a destroy HGFS session protocol request any folder
the outstanding notifications for the session should be removed.
Since that currently does not happen, e.g. when an HGFS session is destroyed via the
protocol requests from a client and goes on to create a new one, during
a Windows VM reboot, then when host updates do occur on an HGFS share the
notifications will be generated that hold pointers to invalid sessions.
This causes the VMX to crash.
The fix is simply to move the notification teardown for a session is at the session
teardown function, from the transport session disconnect.
. File locking: tolerate another race in FileUnlockIntrinsic
. Add util function for getting the epoch from a TimeUtil_Date
. Add LRO defines for VMXNET3
. changes in shared code that don't affect open-vm-tools functionality
The HGFS protocol function to get attributes of a file or folder does
a Posix open without the O_NONBLOCK flag. This can cause the open to block
if there isn't anyone on the other end of the pipe.
Fix is to use the same flags we use for the protocol open to ensure the
correct basic flags are used. Also, if the share is not allowed to follow
symlinks we get this flag set correctly too now.
Begin to consolidate the client to server and server to client VMCI transport
headers to be just one common header for both.
Currently, there are two one for each direction and they are almost
identical. Also the header for the client to the server does not
have all information that is required for asynchronous packet sending
and replies. Hence the motivation for improving this.
The packet headers are poorly defined as they do not have any reserved
fields for extending the current version and they are inconsistent with
the packet types with each direction.
This initial change pulls out the version and packet type into a header
node that will be common to the version one and newer versions of the
transport header.
This involves modifying the server and client sides. The version 1 transport
header will remain binary compatible with this change, it just moves the
first two fields into a common substructure.
Broke out the transport header to return the information from multiple
versions of the header.
Moved the validate datagram function call to the main receive message callback.
Removed the transport header check from this function too, as it is not
useful to have it there. It will be done by each of the specific header version
handling routines.
OVT: require glib2 >= 2.14 and disable 'deprecated-declarations' warnings
Internally we are using 2.14.2 on FreeBSD, 2.16.4 on Solaris, and
2.24.2 on Linux, so let's stop pretending that we work with earlier
versions. RHEL5 with 2.12 hasn't been compiling for a while and even
if it would it would have reduced functionality as glib_regexp would
not be available. FreeBSD started packaging glib2 2.22 starting with 7.1.
Also, mutex API in Glib2 is a mess, they keep changing and deprecating
it, so for now simply add -Wnoerror=deprecated-declarations to avoid
build errors.
Remove some stuff that made no sense. Get reply packet set the pointers
for the reply and then called GetBuf which just returns the pointer
passed in. So I removed the GetBuf call. Cleaned up more unused crap.
HGFS: clean up the transport channel callbacks from the HGFS server
The transport channel callbacks for the HGFS packet request-reply to read,
write and release the memory use badly defined arguments. It uses character
pointer and calls it a token. The type is simply just wrong and should be
treated as an opaque type owned and manipulated only by the channel transport.
Secondly, it is really a context although it is a token, it could be anything
the channel chooses and could change in the future.
This change fixes these issues to be a void *context. This shows that the type
and meaning are not to be exposed beyond the owner of object which is the channel
in this case.
There is no reason to open the device node for writes because we only
ever call ioctl(2) on it and none of the ioctls require write
permissions. This allows our more security concious customers to
restrict the permissions on the device without breaking functionality.
Allow vSockets to the VMX to survive a context ID change.
On vMotion there is an (unlikely) possibility that the context ID of the
virtual machine will conflict with a VM that is already running on the
destination host. To prevent this from distrupting communication a few
changes are necessary.
- The entire VMCI queue pair handle must be checkpointed, allowing the
queue pair to still be used.
- The guest driver must assume that if the context ID it receives as
the destination of a VMCI datagram is not the same as the one it
was expecting then its context ID must have changed and it should
update it. This assumption is valid because the hypervisor will never
deliver a datagram with the wrong context ID.
- To make the new test-vmx test work the range check in VMCI.SetID
had to be removed. Vigor doesn't support unsigned integers so a
negative value for the new context ID is valid.
A new version of the ttylinux iso is included containing a proposed
version of this patch for upstreaming.
Call getsockname(2) once the connection has succeeded to record the
local socket address in the AsyncSocket struct so that
AsyncSocket_GetLocalVMCIAddress works for sockets using the kernel vsock
driver as well.
This allows me to print the local socket address in vsockTest, which
is useful for debugging.
Skip a lot of extraneous logic in VSockSocket_Send and VSockSocket_Recv
by using vmci_qpair_enqueue and vmci_qpair_dequeue directly. This is the
same idea as what happens on Mac OS X when data is copied into the host
pipe in VSockOS_HandleStreamRecv. This allows us to copy data out of the
queue pair as quickly as possible and flush send buffers into the queue
pair without waiting for a callback on the poll thread. Client callbacks
must still be fired on the poll thread to preserve the AsyncSocket API.
HGFS: clean up the VMCI mapping of transport status from the packet
The mapping was a little messy and made some assumptions which we get
away with but should be made more robust.
The first IOV length was never correctly verified against the size
of the transport status and if it was contained completely in the first
page. There was an assumption that this was always the case but the
Windows client would send pings containing IOVs of buffers on the stack
and as such could easily break the VMX.
The adjust of the packet IOVs by the VMCI code to skip over the VMCI
transport status before passing to the HGFS server to process has also
been cleaned up. As has the restoration of the packet IOVs on final
processing to access the packet VMCI transport status to set complete.
In order to simplify this the HgfsPacket structure now contains an IOV
field for the transport channel to use which holds the start of the
VMCI channel transport status IOVs. This is then used to restore once
the HGFS server has completed processing and the transport channel is
doing the final send processing.
Export local and remote VMCI address info from AsyncSocket.
I need this information to look up vSocket IDs after a resume,
but this will also be necessary to determine if the peer is
a privileged socket (bound to a port less than 1024).
Checkpoint routines for AsyncSockets backed by VSockSocket.
These checkpoint routines work a little differently than those for
VSockSocket. While the vSocket infrastructure checkpoints all sockets
which are open (because they are in the global VSockSocketStreamGlobal
tables) each AsyncSocket must be checkpointed by its owner. This gives
the owner the opportunity to reinstall all the necessary callbacks on
resume.
Remove the HgfsVaIov and the unused HSPU_GetDataPacketIov that used it.
If we needed it is not clear and what the rationale for any expected
usage was not commented. So removed it.
Dis alba VMCI socket call in guestRPC completely to avoid triggering
unstable code. Even guestRPC vmci usage is disabled on VMX side, tools
will still try to make a connection and then fall back to use backdoor
if the connection fails. This triggers potential issues. Will do more
testing and lift this.
This change determines if the VM's virtual disk is on SSD and make this
information available to guest. This is already implemented on ESX and
Mac OS. This change implements it on Windows and Linux.
On Linux rotational property of the disk and block device corresponding
to logical volume can be read from /sys/block/dev-name/queue/rotational
Integration with VMCI CID 0 changes for polling reduction in guestRPC.
Hookup changes to VMCI changes for polling reduction in guestRPC by
calling new API to listen on VMX side. Without this change, we can only
create one guestRPC listening socket per host, with this change, we can
create one guestRPC listening socket per VM.
Reduce polling in guestRpc TCLO messaging. GuestRpc is using backdoor, the
guest has to poll periodically to check request availability. Replacing
backdoor using vsocket allows us to get rid of this polling completely.
HGFS: Clean up server of the reply packet handling
The reply packet handler was returning a bool when in fact it cannot
fail. Fix this up and make it consistent with other calls for the reply
packet handler when filling the header in the complete request common
function.
This is a lot of repeating the same change for all the opcodes of each
supported request and version.
HGFS: we shouldn't allow-open on blank hostPaths for shares by default
Any user can edit the VMX file and set any host path of a shared folder
to the empty string. For cases where the host path is an empty string
it will cause every host drive to be shared with the guest VM.
This should not be allowed to occur by default in case a user mistakenly
sets or maliciously sets the string to empty. It should only be allowed
when a user is intending that behavior and understands the potential
issues.
To fix this I have added an additional VMX config file setting that a user
would have to explicitly set to enable this feature and set a shared folder
to an empty string.
VMCI: switch to upstreamed Linux VMCI API for internal users, pt 2
Second part of the change to switch to the upstreamed API. This one
tweaks the API for Linux only so that it matches exactly the one that
is upstreamed.
This change uses Haswell HLE in our mutex locks. HLE is Hardware
Lock Elision, which are hints to the processor to start/end a
hardware transaction. It is enabled via:
lock acquire REPNZ 0xf2
lock release REPZ 0xf3
On machines without HLE, these prefixes are ignored by the hardware.
Fix VMCISock_GetAFValue() for upstream vsocket testing
The Tools driver exposes an IOCTL to get the address family, since
it's not fixed. The upstream driver was given a fixed address family,
so it doesn't need this IOCTL. But our tests (and other apps) still
try to use the IOCTL, which causes them to fail with the upstream
driver. Fix our header (the Tools version) so that it can cope with
both.
Note: not very performant, so best to cache the result if possible.
The test suite common code already caches it.
When VMCI_EVENT_VALID_VMX was changed to accept the QP_PEER_ATTACH and
QP_PEER_DETACH events this broke the regular non-VMX version. This
wasn't caught because I didn't rebuild my guest VMCI driver when
testing.
HGFS: clean up server packet information duplication
Each packet sets a bool in the HgfsPacket object to state whether the
packet can be processed asynchronously by this transport.
This data is not required and doesn't make sense to duplicate this for
every packet received by the transport. The same information is sent to
the HGFS server at connect time from the transport. This information is
part of the transport characteristics and the server already makes use
of some of these flags, e.g. shared memory support for some operation
requests like change notification.
Replace the packet bool with the correct transport characteristics flag
check for asynchronous packet handling capabilities.
Emit attach and detach events when a queue pair peered with the
hypervisor context ID is alloced or detached by the guest.
Now that I understand these events better I have restored the validity
checks and made the attach and detach events valid in the VMX. The
QP resumed event is still not relevant in the VMX so the vsock layer
should not subscribe to it.
HGFS: add new flags to Hgfs create session request
The Hgfs create session request assumes currently that the only flags
required are on a per request type basis. This is simply not true. There
are some features that trancend many operations and will require a
general per session flag to denote support.
Two examples that immediately come to mind: oplocks and short name
support.
For oplocks the server is going to need to know ahead of any requests
that the client is going to be using this feature if supported by the
server. Furthermore, oplocks affect the whole operation of requests as
far as the server is concerned. The server should handle requests
asynchronously if oplocks are in play to prevent deadlock. Also, oplocks
are taken out a file open time but can be broken by many other requests
not just concurrent opens.
Also for short name support for the host, these can occur not just with
directory listings and enumeration but for the client opens too.
Currently, the client must know if the server does support short names
so that it does not try and look up the short name itself.
Use the common VMCI socket driver code in the VMX to provide access to
raw VMCI sockets. This functionality has some limitations, for example
blocking calls are not possible because blocking a VMX thread can
prevent the guest from making the necessary progress and cause deadlock.
A new lock parameter has been added to VSockSocket_Socket to allow the
caller to provide a lock for use by the socket. This is similar to the
lock provided to AsyncSocket and allows an AsyncSocket and VSockSocket
to share the same lock.
VMCI_DatagramSend requires a needsLock parameter. This is currently
hardcoded to FALSE in all cases. This may not be correct and needs to be
analyzed.
HGFS: cleanup server request and header unpacking part III
The complete request processing which processes just packing the Hgfs
header for the completing request needs cleaning up. It can be
simplified and more processing done in a common pack header reply
routine for both legacy (old header) and new header with session enabled
requests.
Also the input params teardown should be moved into its own specialized
routine to complement the init routine in Part II of this mini series.
HGFS: cleanup server request and header unpacking part II
After the first cleanup and extracting the header details by individual
routines and now we have the session information and routines separated.
1) Cleanup the unpack params routine:
- Make it just do the unpacking of the headers and validate the fields.
- Make any bad header sizes or incorrect values of the header fiels untrusted
data and we drop the packet by returning internal error. It makes no sense
to try and reply if the data in the header is not valid.
- Move the transport/server Hgfs packet handling, the session info and the
input params back to the hgfs server main file into a parent function of the
unpack params. The new parent function HgfsServerGetRequest is called from
the server receive message routine now instead.
- Make the input request handling variables const as the input data is not
modified.
2) Cleanup the processing a little more in the new HgfsServerGetRequest
function by doing the can fail processing first: Hgfs Packet, and unpack
packet params first. If we have valid data, determine the session and
then allocate and set the input params. Saves a potential processing
with the input params not initialized to anything but zero.
3) Remove the useless and duplicated function HgfsValidatePacket.
4) Move HgfsPackReplyHeaderV4 function declaration to the correct header file.
. lib/file: cap the max supported file size
. use special call instead of slower statfs to get FS type;
. product version changes
. a small cleanup in DnD code
. changes in shared code that don't affect open-vm-tools functionality
VSOCK: assign a new resource ID for hypervisor stream sockets.
The resouce ID used for VMCI Socket control packets (0) is already
used for the VMCI_GET_CONTEXT_ID hypercall so a new RID (15) must
be used when the guest sends these datagrams to the hypervisor.
This resource ID replaces VMCI_RPC_PRIVILEGED which is unused in
shipping products. VMCI_RPC_UNPRIVILEGED is also removed.
This change also enables VMCI Socket communication to the hypervisor
by removing CID 0 from internal blacklists.
HGFS: cleanup server request and header unpacking part I
The Hgfs server HgfsParseRequest routine to unpack packet header and
validation is somewhat tangled and convoluted. This attempts to clean
that up and make way for checking for the header flags if available
and any other information that maybe passed from the client on a per
request basis in the future.
The changes include:
- rename the HgfsParseRequest to HgfsUnpackPacketParams which is more
correct and consistent with the naming scheme for HGFS unmarshalling of
arguments.
- Split out the unpacking of the different protocol versions of header
and packet details into their respective subroutines thus simplifying
the main routine itself.
- Split out the HGFS session lookup and/or creation that occurs as part
of this routine into a subroutine call.
This cleans up some ordering issues of packet validation and data
extraction from that packet. It helps clean up some of the session
lookup which should be handled separately from this function which is
server specific and not protocol packet definition. A later change set
to deal with that.
projects / thirdparty / open-vm-tools.git / log
