From: Andrew Tridgell Date: Wed, 20 May 2026 04:57:50 +0000 (+1000) Subject: import rsync-web website content as a subdirectory X-Git-Url: http://git.ipfire.org/gitweb/index.cgi?a=commitdiff_plain;h=0af88421dc65dfade55f0f6fc59a4c2edc2918b8;p=thirdparty%2Frsync.git import rsync-web website content as a subdirectory Fold the standalone rsync-web repo into the rsync source tree as rsync-web/, eliminating the sibling-checkout convention and the drift it causes between the release-time HTML snapshot in ../release/rsync-html and the source of truth in ../rsync-web. Flat-copy import (no git history merge). The standalone repo at github.com/RsyncProject/rsync-web is retained for historical reference and will be archived once the in-tree copy proves itself. Add /rsync-web/ to .gitattributes with export-ignore so the website content does not bloat the release source tarball produced by 'git archive' in packaging/release.py step_7_tarball. A follow-up commit repoints HTML_SRC in packaging/release.py at the new in-tree location. --- diff --git a/.gitattributes b/.gitattributes index 6313b56c..dd440b78 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1 +1,8 @@ * text=auto eol=lf + +# The rsync-web/ subdirectory holds the project website source content +# (mirrors what gets pushed to https://rsync.samba.org). Exclude it from +# `git archive` output so the release source tarball produced by +# packaging/release.py step_7_tarball does not bloat with HTML the +# tarball doesn't need. +/rsync-web/ export-ignore diff --git a/rsync-web/.gitignore b/rsync-web/.gitignore new file mode 100644 index 00000000..5a606d7c --- /dev/null +++ b/rsync-web/.gitignore @@ -0,0 +1,11 @@ +/.xvpics +/doxygen/ +/tech_report/IMG_PARAMS.dir +/tech_report/IMG_PARAMS.pag +/upload +/netware +/pre-change +/index.html-* +/rsync-and-debian/rsync-and-debian.html +/rsync-and-debian/rsync-and-debian.ps +/badge.svg diff --git a/rsync-web/COPYING.html b/rsync-web/COPYING.html new file mode 100644 index 00000000..797f6cb4 --- /dev/null +++ b/rsync-web/COPYING.html @@ -0,0 +1,674 @@ + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/> + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + <program> Copyright (C) <year> <name of author> + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +<https://www.gnu.org/licenses/>. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +<https://www.gnu.org/philosophy/why-not-lgpl.html>. diff --git a/rsync-web/COPYING2.html b/rsync-web/COPYING2.html new file mode 100644 index 00000000..5c38bb12 --- /dev/null +++ b/rsync-web/COPYING2.html @@ -0,0 +1,340 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc. + 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Library General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. +
+ GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) +
+These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. +
+ 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. +
+ 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS +
+ How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + <signature of Ty Coon>, 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Library General +Public License instead of this License. diff --git a/rsync-web/FAQ.html b/rsync-web/FAQ.html new file mode 100644 index 00000000..7368a8a8 --- /dev/null +++ b/rsync-web/FAQ.html @@ -0,0 +1,284 @@ + + + +rsync FAQ + + + +

Frequently Asked Questions

+ +
    
+ +
+

the transfer fails to finish

+ +

If you get an error like one of these: + +

rsync: error writing 4 unbuffered bytes - exiting: Broken pipe
+rsync error: error in rsync protocol data stream (code 12) at io.c(463)
+
+ +

or + +

rsync: connection unexpectedly closed (24 bytes read so far)
+rsync error: error in rsync protocol data stream (code 12) at io.c(342)
+
+ +

please read the issues and debugging page +for details on how you can try to figure out what is going wrong. + +


+

rsync recopies the same files

+ +

Some people occasionally report that rsync copies too many files when +they expect it to copy only a few. In most cases the explanation is +that you forgot to include the --times (-t) option in the original copy, +so rsync is forced to (efficiently) transfer every file that differs in +its modified time to discover what data (if any) has changed. + +

Another common cause involves sending files to an Microsoft filesystem: +if the file's modified time is an odd value but the receiving filesystem +can only store even values, then rsync will re-transfer too many files. +You can avoid this by specifying the --modify-window=1 option. + +

Yet another periodic case can happen when daylight-savings time +changes if your OS+filesystem saves file times in local time instead of +UTC. For a full explanation of this and some suggestions on how to +avoid them problem, see this document. + +

Something else that can trip up rsync is a filesystem changeing the +filename behind the scenes. This can happen when a filesystem changes +an all-uppercase name into lowercase, or when it decomposes UTF-8 behind +your back. + +

+ +

An example of the latter can occur with HFS+ on Mac OS X: if you +copy a directory with a file that has a UTF-8 character sequence in it, +say a 2-byte umlaut-u (\0303\0274), the file will get that character +stored by the filesystem using 3 bytes (\0165\0314\0210), and rsync will +not know that these differing filenames are the same file (it will, in +fact, remove a prior copy of the file if --delete is enabled, and then +recreate it). + +

You can avoid a charset problem by passing an appropriate --iconv +option to rsync that tells it what character-set the source files are, +and what character-set the destination files get stored in. For +instance, the above Mac OS X problem would be dealt with by using +--iconv=UTF-8,UTF8-MAC (UTF8-MAC is a pseudo-charset recognized by Mac +OS X iconv in which all characters are decomposed). + +

+ +

If you think that rsync is copying too many files, look at the +itemized output (-i) to see why rsync is doing the update (e.g. the 't' +flag indicates that the time differs, or all pluses indicates that rsync +thinks the file doesn't exist). You can also look at the stats produced +with -v and see if rsync is really sending all the data. See also the +--checksum (-c) option for one way to avoid the extra copying of files +that don't have synchronized modified times (but keep in mind that the +-c option eats lots of disk I/O, and can be rather slow). + +


+

is your shell clean

+ +

The "is your shell clean" message and the "protocol mismatch" message +are usually caused by having some sort of program in your .cshrc, .profile, +.bashrc or equivalent file that writes a message every time you connect +using a remote-shell program (such as ssh or rsh). Data written in this +way corrupts the rsync data stream. rsync detects this at startup and +produces those error messages. However, if you are using rsync-daemon +syntax (host::path or rsync://) without using a remote-shell program (no +--rsh or -e option), there is not remote-shell program involved, and the +problem is probably caused by an error on the daemon side (so check the +daemon logs). + +

A good way to test if your remote-shell connection is clean is to try +something like this (use ssh or rsh, as appropriate): + +

ssh remotesystem /bin/true > test.dat
+ +

That should create a file called test.dat with nothing in it. If +test.dat is not of zero length then your shell is not clean. Look at the +contents of test.dat to see what was sent. Look at all the startup files on +remotesystem to try and find the problem. + +


+

memory usage

+ +

Rsync versions before 3.0.0 always build the entire list of files to be +transferred at the beginning and hold it in memory for the entire run. Rsync +needs about 100 bytes to store all the relevant information for one file, +so (for example) a run with 800,000 files would consume about 80M of +memory. -H and --delete increase the memory usage further. + +

Version 3.0.0 slightly reduced the memory used per file by not storing fields +not needed for a particular file. It also introduced an incremental recursion +mode that builds the file list in chunks and holds each chunk in memory only as +long as it is needed. This mode dramatically reduces memory usage, but it +only works provided that both sides are 3.0.0 or newer and certain options that +rsync currently can't handle in this mode are not being used. + +


+

out of memory

+ +

The usual reason for "out of memory" when running rsync is that you are +transferring a _very_ large number of files. The size of the files doesn't +matter, only the total number of files. If memory is a problem, first try to +use the incremental recursion mode: upgrade both sides to rsync 3.0.0 or +newer and avoid options that disable incremental recursion (e.g., use +--delete-delay instead of --delete-after). If this is not +possible, you can break the rsync run into smaller chunks operating on +individual subdirectories using --relative and/or exclude rules. + +


+

rsync through a firewall

+ +

If you have a setup where there is no way to directly connect two +systems for an rsync transfer, there are several ways to get a firewall +system to act as an intermediary in the transfer. You'll find full details +on the firewall methods page. + +


+

rsync and cron

+ +

On some systems (notably SunOS4) cron supplies what looks like a socket +to rsync, so rsync thinks that stdin is a socket. This means that if you +start rsync with the --daemon switch from a cron job you end up rsync +thinking it has been started from inetd. The fix is simple—just +redirect stdin from /dev/null in your cron job. + +


+

rsync: Command not found

+ +

This error is produced when the remote shell is unable to locate the rsync +binary in your path. There are 3 possible solutions: + +

    + +
  1. install rsync in a "standard" location that is in your remote path. + +
  2. modify your .cshrc, .bashrc etc on the remote system to include the path +that rsync is in + +
  3. use the --rsync-path option to explicitly specify the path on the +remote system where rsync is installed + +
+ +

You may echo find the command: + +

ssh host 'echo $PATH'
+ +

for determining what your remote path is. + +


+

spaces in filenames

+ +

Can rsync copy files with spaces in them? + +

Short answer: Yes, rsync can handle filenames with spaces. + +

Long answer: + +

Rsync handles spaces just like any other unix command line application. +Within the code spaces are treated just like any other character so a +filename with a space is no different from a filename with any other +character in it. + +

The problem of spaces is in the argv processing done to interpret the +command line. As with any other unix application you have to escape spaces +in some way on the command line or they will be used to separate arguments. + +

It is slightly trickier in rsync (and other remote-copy programs like +scp) because rsync sends a command line to the remote system to launch the +peer copy of rsync (this assumes that we're not talking about daemon mode, +which is not affected by this problem because no remote shell is involved +in the reception of the filenames). The command line is interpreted by the +remote shell and thus the spaces need to arrive on the remote system +escaped so that the shell doesn't split such filenames into multiple +arguments. + +

For example: + +

rsync -av host:'a long filename' /tmp/
+ +

This is usually a request for rsync to copy 3 files from the remote +system, "a", "long", and "filename" (the only exception to this is for a +system running a shell that does not word-split arguments in its commands, +and that is exceedingly rare). If you wanted to request a single file with +spaces, you need to get some kind of space-quoting characters to the remote +shell that is running the remote rsync command. The following commands +should all work: + +

rsync -av host:'"a long filename"' /tmp/
+rsync -av host:'a\ long\ filename' /tmp/
+rsync -av host:a\\\ long\\\ filename /tmp/
+ +

You might also like to use a '?' in place of a space as long as there +are no other matching filenames than the one with spaces (since '?' matches +any character): + +

rsync -av host:a?long?filename /tmp/
+ +

As long as you know that the remote filenames on the command line +are interpreted by the remote shell then it all works fine. + +


+

ignore "vanished files" warning

+ +

Some folks would like to ignore the "vanished files" warning, which +manifests as an exit-code 24. The easiest way to do this is to create +a shell script wrapper. For instance, name this something like +"rsync-no24": + +

#!/bin/sh
+rsync "$@"
+e=$?
+if test $e = 24; then
+    exit 0
+fi
+exit $e
+ +
+

read-only file system

+ +

If you get "Read-only file system" as an error when sending to a rsync +daemon then you probably forgot to set "read only = no" for that module. + +


+

multiplexing overflow 101:7104843

+ +

This mysterious error, or the similar "invalid message 101:7104843", can +happen if one of the rsync processes is killed for some reason and a message +beginning with the four characters "Kill" gets inserted into the protocol +stream as a result. To solve the problem, you'll need to figure out why rsync +is being killed. + +


+

inflate (token) returned -5

+ +This error means that rsync failed to handle an expected error from the +compression code for a file that happened to be transferred with a block size +of 32816 bytes. You can avoid this issue for the affected file by transferring +it with a manually-set block size (e.g. --block-size=33000), or by upgrading +the receiving side to rsync 3.0.7. + +
+ + diff --git a/rsync-web/GPL.html b/rsync-web/GPL.html new file mode 100644 index 00000000..c734dfd2 --- /dev/null +++ b/rsync-web/GPL.html @@ -0,0 +1,16 @@ + + + +rsync's license + + + +Beginning with 3.0.0, rsync is available under the GNU General Public +License version 3. (Older releases were available under the +GPL version 2.) + +

+
+
+ + diff --git a/rsync-web/GPL2.html b/rsync-web/GPL2.html new file mode 100644 index 00000000..b97abe83 --- /dev/null +++ b/rsync-web/GPL2.html @@ -0,0 +1,15 @@ + + + +rsync's license + + + +Releases prior to 3.0.0 were released under the +GNU General Public License version 2: + +

+
+
+ + diff --git a/rsync-web/backup.txt b/rsync-web/backup.txt new file mode 100644 index 00000000..c09fb57d --- /dev/null +++ b/rsync-web/backup.txt @@ -0,0 +1,36 @@ +#!/bin/sh + +# This script does personal backups to a rsync backup server. You will end up +# with a 7 day rotating incremental backup. The incrementals will go +# into subdirectories named after the day of the week, and the current +# full backup goes into a directory called "current" +# tridge@linuxcare.com + +# directory to backup +BDIR=/home/$USER + +# excludes file - this contains a wildcard pattern per line of files to exclude +EXCLUDES=$HOME/cron/excludes + +# the name of the backup machine +BSERVER=owl + +# your password on the backup server +export RSYNC_PASSWORD=XXXXXX + + +######################################################################## + +BACKUPDIR=`date +%A` +OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES + --delete --backup --backup-dir=/$BACKUPDIR -a" + +export PATH=$PATH:/bin:/usr/bin:/usr/local/bin + +# the following line clears the last weeks incremental directory +[ -d $HOME/emptydir ] || mkdir $HOME/emptydir +rsync --delete -a $HOME/emptydir/ $BSERVER::$USER/$BACKUPDIR/ +rmdir $HOME/emptydir + +# now the actual transfer +rsync $OPTS $BDIR $BSERVER::$USER/current diff --git a/rsync-web/bar1.jpg b/rsync-web/bar1.jpg new file mode 100644 index 00000000..7c6acf3c Binary files /dev/null and b/rsync-web/bar1.jpg differ diff --git a/rsync-web/bin/badge-update b/rsync-web/bin/badge-update new file mode 100755 index 00000000..2b5b72ad --- /dev/null +++ b/rsync-web/bin/badge-update @@ -0,0 +1,4 @@ +#!/bin/sh +rm -f badge.svg +wget https://github.com/RsyncProject/rsync/workflows/build/badge.svg +rsync -aiic --inplace --remove-source-files badge.svg $SAMBA_HOST:/home/httpd/html/rsync/ diff --git a/rsync-web/bin/upload b/rsync-web/bin/upload new file mode 100755 index 00000000..39bae3e5 --- /dev/null +++ b/rsync-web/bin/upload @@ -0,0 +1,7 @@ +#!/bin/sh +if [ -d rsync-and-debian ]; then + rsync -aviOHFFc --del -f._filt . $SAMBA_HOST:/home/httpd/html/rsync/ "${@}" +else + echo "Run this from the root of the html hierarchy." + exit 1 +fi diff --git a/rsync-web/bug-tracking.html b/rsync-web/bug-tracking.html new file mode 100644 index 00000000..92e4c84a --- /dev/null +++ b/rsync-web/bug-tracking.html @@ -0,0 +1,69 @@ + + + +rsync bug-tracking + + + +

rsync bug-tracking

+ +

Please use this checklist combined with the help on the +issues and debugging page before +reporting a bug. Thanks! + +

+ +

To report a bug or make suggestions, use one of these methods: + +

+ +

Thanks for helping out! + + diff --git a/rsync-web/bugtracking.html b/rsync-web/bugtracking.html new file mode 100644 index 00000000..351087c0 --- /dev/null +++ b/rsync-web/bugtracking.html @@ -0,0 +1,7 @@ + + + +rsync bug-tracking + + + diff --git a/rsync-web/bugzilla.html b/rsync-web/bugzilla.html new file mode 100644 index 00000000..351087c0 --- /dev/null +++ b/rsync-web/bugzilla.html @@ -0,0 +1,7 @@ + + + +rsync bug-tracking + + + diff --git a/rsync-web/convert-gpl b/rsync-web/convert-gpl new file mode 100644 index 00000000..463d22d4 --- /dev/null +++ b/rsync-web/convert-gpl @@ -0,0 +1,11 @@ +#!/usr/bin/perl +use strict; + +while (<>) { + s/&/&/g; + s//>/g; + s/ /


/; + s#(<)(https?:.*?)(>)#$1$2$3#g; + print $_; +} diff --git a/rsync-web/daylight-savings.html b/rsync-web/daylight-savings.html new file mode 100644 index 00000000..8c74b641 --- /dev/null +++ b/rsync-web/daylight-savings.html @@ -0,0 +1,183 @@ + +DST change and date comparisons + + +

J.W. Schultz wrote the following text in a +post +to the cygwin mailing list.  It has been slightly edited and beautified +for inclusion here. + +

How the DST Change can adversely affect FAT filesystems

+ +

+The vernal and autumnal transitions to and from daylight-savings +time have important implications +for those with Microsoft systems and use utilities that +compare file timestamps on different filesystem types or +with filesystems on other operating systems that can lead to +a problem in how the file's date is handled. +This problem lies in the way FAT filesystems stores +timestamps and how Windows converts between local time and +UTC. + +

Background

+ +

+In UNIX and UNIX-like systems (such as Linux) file timestamps +are stored in UTC (universal time) and are only converted to +local-time by user-space programs for display purposes.  At +the system call level all time values are in UTC and +utilities that compare timestamps do so in UTC.  Also, the +standard UTC->local and local->UTC conversion functions are +aware of DST and conversions reflect this so that if a +timestamp was recorded during ST it will be converted using +the ST offset even when the current system time is DST. + +

+In Windows things are not so simple.  Windows operates in +local-time.  Timestamps in the various FAT derived +filesystems are stored in local-time.  Timestamps in NTFS +filesystems are stored in UTC.  This inconsistency is +further complicated by the fact that the conversion routines +used are not DST aware.  Instead of being DST aware the +system has a fixed offset to convert between local-time and +UTC regardless of the date in the timestamp.  This fixed +offset is calculated at boot time and only changed when +systems transition to or from DST.  As a result the apparent +modification time of a file on NTFS as reported in a windows +utility will change by one hour when reported in local-time +and FAT based files when reported in UTC. + +

+The difficulty that this produces is that any utilities that +compare timestamps between FAT and NTFS filesystems or +between Windows and other platforms will view files that +have not changed as having a different modified time.  Among other things +this will affect rsync, rdiff, unison, wget, and make.  However, +for the purposes of this document, we will only discuss rsync. + +

+With the reduced cost of hard disks many newer backup +systems are using hard disk based storage and take advantage +of timestamp comparison to detect file changes for the sake +of efficiency.  Rsync is probably premier in this role and +is used by a fair number of free and even commercial backup +systems as well as being the basis for many home-brew backup +solutions. + +

+With rsync and similar systems the effect of this is that +every file will appear to have been changed.  The result is +any space savings associated with linking (--link-dest) or +with decremental backup approaches (--compare-dest and +--backup-dir) will be defeated.  Perhaps worse, because +every file will appear to have changed the time required to +do a backup or a non-backup rsync will be much longer than +normal.  In some cases backups that normally complete in +less than one hour can take several days. + +

+So what can be done about it? Several things, there are +ways to merely mitigate the problem, to correct it and finally +to prevent the problem entirely. + +

Mitigation

+ +

+Rsync has a --modify-window option.  Many of you already use +--modify-window=1 to cope with the fact that windows often +stores timestamps with a two second resolution.  Using a +--modify-window=3601 will cause rsync to ignore timestamp +differences of up to one hour. + +

+This if often not particularly dangerous because a file would have to +be changed, synced and changed again without changing size +within a single hour and have no subsequent changes for this copy +to miss a file change.  However, for some systems any such risk is +unacceptable, so other solutions are needed. + +

Correcting the Timestamps

+ +

+There are two ways to correct. + +

For the first run after the time change, you can run rsync with the +--checksum option in order to ensure that only files that have a changed +checksum get transferred, and update the modified time on all unchanged +files.  This option has the drawback that it increases disk I/O by +a large amount on both the sending and receiving side, slowing down the +copy. + +

+The other way to correct things is to change the timestamps +on the files on the backup server before doing a copy after a +time change has occurred. + +

+Included here is an example perl script that will change the +timestamps of files in a list on standard-input.  Whether +you use a positive or negative shift will depend on which +end you decide to adjust. + +

+This is an example of how to use the script: + +

+touch -d '01:00 13-apr-03' /tmp/cmpfile
+find . -type f ! -newer /tmp/cmpfile | shifttime.pl 3600
+
+ +
+
+#!/usr/bin/perl
+use strict;
+
+my $offset = shift() + 0;
+die "Usage: $0 OFFSET_SECONDS\n" unless $offset;
+
+while (<>) {
+    chomp;
+    my $mtime = (stat $_)[9];
+    next unless $mtime;
+    $mtime += $offset;
+    utime $mtime, $mtime, $_;
+}
+
+
+ +

Prevention

+ +

+To prevent the problem in the first place you need to +prevent changing to DST.  This can be done by either running +the windows system in UTC, by disabling DST and changing +the system time manually twice each year, or by avoiding the use +of the FAT filesystem (perhaps by switching to a different OS). + + +

Notes and References

+ +

+Here are some references that Wayne Piekarski collected +while researching this problem.  They contain a lot of +information about the ways that Windows deals with +timestamps on NTFS and FAT filesystems. + +

+http://optics.ph.unimelb.edu.au/help/rsync/rsync_pc1.html#gotchas
+http://list-archive.xemacs.org/xemacs-nt/199911/msg00130.html
+http://p2p.wrox.com/archive/c_plus_plus_programming/2001-06/53.asp
+http://www.codeproject.com/datetime/dstbugs.asp
+http://support.microsoft.com/default.aspx?scid=kb;[LN];158588 + +

+I wish to thank Wayne Piekarski for having copiled the +references and also supplying some additional insights. + +

+Permission is granted without reservation reprint and +distribute this in whole and in part to any interested +parties. + +

—J.W. Schultz diff --git a/rsync-web/doc-resources.html b/rsync-web/doc-resources.html new file mode 100644 index 00000000..e9e0bd5f --- /dev/null +++ b/rsync-web/doc-resources.html @@ -0,0 +1,15 @@ + + +

  • A nice tutorial on setting up ssh to avoid password prompts. + +
  • Karsten Thygesen has written a doc on how to setup +anonymous rsync servers. + +
  • Anthony Wesley has written a doc on +how to build rsync for Windows95. + +
  • Mike McHenry has written up some info on how to get rsync working under NT. + +
  • Michael Holve has written a very useful rsync tutorial. + +
  • Daniel Teklu has a detailed HOWTO on rsync and stunnel. diff --git a/rsync-web/documentation.html b/rsync-web/documentation.html new file mode 100644 index 00000000..6cb5aad6 --- /dev/null +++ b/rsync-web/documentation.html @@ -0,0 +1,54 @@ + + + +rsync documentation + + + +

    documentation

    + + + +

    (Some of the above items are also listed on the rsync resources page. + + diff --git a/rsync-web/download.html b/rsync-web/download.html new file mode 100644 index 00000000..b931d938 --- /dev/null +++ b/rsync-web/download.html @@ -0,0 +1,127 @@ + + + +rsync download + + + +

    rsync download

    + +
    + + +
    + +

    Source-code releases

    + +

    You can grab the latest source code and other related files in a variety of ways: + +

    + +Once you have the source, read the INSTALL.md file for +details on some development libraries that you will need to build it. + +

    The GPG Signing Key

    + +The GPG signing key that is used to sign the release files is available from the public pgp key-server +network. If you have automatic key-fetching enabled, just running a normal +"gpg --verify" will grab my key automatically. +Or, feel free to grab the gpp + key for Wayne Davison manually.

    + +From 3.4.0 and later releases will be signed by Andrew +Tridgell. Please fetch the key for andrew@tridgell.net from https://keys.openpgp.org/ + +

    Binaries

    + +

    Precompiled binaries are available in most modern OS distributions, so +you should first check if you can install an rsync package via your +standard package-install tools for your OS. + +

    Ubuntu

    + +

    The rsync project maintains a Launchpad PPA that tracks upstream stable +releases for the currently supported Ubuntu series (jammy 22.04 LTS, +noble 24.04 LTS, questing 25.10, resolute 26.04 LTS). This is the +fastest way to get the latest upstream rsync on Ubuntu without waiting +for the distro to update its packaged version: + +

    sudo add-apt-repository ppa:rsyncproject/rsync
    +sudo apt update && sudo apt install rsync
    + +

    See the PPA page on Launchpad +for build status across architectures. + +

    The GitHub Actions +page has build events that each generate a few binary artifact zip files +(just click through via the build's title to see them). The actions page is +also linked via the various green build-status icons on the web pages here. +These builds use the newest libraries, such as xxhash checksums and zstd +compression, and are dynamically linked, so you may need to install some +official library packages for your distribution. If you're curious how the +build was done, you can look at the build rules in the "Workflow file" tab. +See the INSTALL.md +file for some package name hints, though you can use the non-devel versions of +the various lib packages and ignore the gcc/autoconf/awk packages. + +

    There are also packages available from some 3rd-parties (note that we +cannot vouch for 3rd parties, so use a source that you trust): + +

    + +

    Git source vs Release files

    + +The release tar files come with a few generated files that are not checked in to git. +These mainly include the man pages and the configure related files. To make use of +the git-derived files you will need autoconf, autoheader, and a version of python3 +that has the commonmark lib (OR cmarkgfm). If you have trouble with setting up the +those required files, you can try running "./prepare-source fetchgen" to grab the +very latest generated files that were created from the latest commit into the master +branch. + +

    Note: Since the source repository is a work in progress it may, at +times, not compile or fail in various ways, though it is usually pretty good. + +

    Source repository patches

    + +

    There are also various patch files in the "rsync-patches" repository that +represent either some work-in-progress features or features that are considered +to be a little too fringe-interest for the main release. See the github link +above for how to look around at what is available, or snag a release tar file. +The maintainer like to put the rsync-patches dir into his rsync checkout as a +directory named "patches" and has some helper scripts for how to use local git +branches to test and update the diffs. + + diff --git a/rsync-web/examples.html b/rsync-web/examples.html new file mode 100644 index 00000000..adaeab7c --- /dev/null +++ b/rsync-web/examples.html @@ -0,0 +1,54 @@ + + + +rsync examples + + + +

    rsync examples

    + +If you have an interesting example of how you use rsync then please +submit it to the +rsync-bugs@samba.org +for inclusion on this page. + +

    backup to a central backup server with 7 day incremental

    + +
    
    +
    +
    + +

    backup to a spare disk

    + +
    
    +
    +
    + +

    mirroring vger CVS tree

    + +
    
    +
    +
    + +

    automated backup at home

    + +
    
    +
    +
    + +

    Fancy footwork with remote file lists

    + +
    
    +One little known feature of rsync is the fact that when run over a
    +remote shell (such as rsh or ssh) you can give any shell command as
    +the remote file list. The shell command is expanded by your remote
    +shell before rsync is called. For example, see if you can work out
    +what this does:
    +
    +	rsync -avR remote:'`find /home -name "*.[ch]"`' /tmp/
    +
    +note that that is backquotes enclosed by quotes (some browsers don't
    +show that correctly).
    +
    + + diff --git a/rsync-web/features.html b/rsync-web/features.html new file mode 100644 index 00000000..34641cef --- /dev/null +++ b/rsync-web/features.html @@ -0,0 +1,28 @@ + + + +rsync features + + + +

    rsync features

    + +rsync is a file transfer program for Unix systems. rsync uses the +"rsync algorithm" which provides a very fast method for bringing +remote files into sync. It does this by sending just the differences +in the files across the link, without requiring that both sets of +files are present at one of the ends of the link beforehand.

    + +Some features of rsync include + +

    + + diff --git a/rsync-web/filt b/rsync-web/filt new file mode 100644 index 00000000..0be6d162 --- /dev/null +++ b/rsync-web/filt @@ -0,0 +1,13 @@ +P /badge.svg +H /.git/ +H /.gitignore +H /bin/ +H /upload/ +P /USE_THE_GIT_rsync-web_REPO_NOW +H /filt +H *.swp +- /netware/ +- /pre-change +- /rsync-and-debian/ +- /index.html-* +- *~ diff --git a/rsync-web/find.html b/rsync-web/find.html new file mode 100644 index 00000000..caf36f3e --- /dev/null +++ b/rsync-web/find.html @@ -0,0 +1,13 @@ + + + +rsync search + + + +

    Search the rsync web pages

    + +

    Searching samba.org will find results in the main web pages and in the +archived mailing-list pages too. + + diff --git a/rsync-web/firewall.html b/rsync-web/firewall.html new file mode 100644 index 00000000..86003dad --- /dev/null +++ b/rsync-web/firewall.html @@ -0,0 +1,189 @@ + + + +rsync firewall tunneling + + + +

    Using rsync through a firewall

    + +

    If you have a setup where there is no way to directly connect two +systems for an rsync transfer, there are several ways to get a firewall +system to act as an intermediary in the transfer. + +

    This first method should work for any remote-shell (e.g. ssh, rsh, etc). +The other methods are all targeted at ssh (which has a lot of flexibility +in making a tunneled connection possible). + +

    Method 1 -- should work with any remote-shell

    + +

    Use your remote shell (e.g. ssh) to access the middle system and have it +use a remote shell to hop over to the actual target system. + +

    To effect this extra hop, you'll need to make sure that the remote-shell +connection from the middle system to the target system does not involve any +tty-based user interaction (such as prompting for a password) because there +is no way for the middle system to access the local user's tty. + +

    One way that should work for all remote-shell programs is to enable host-based +authentication, which would allow all connections from the middle system to +the target system to succeed (when the username remains the same). +However, this may not be a desirable setup. + +

    A better method that works with ssh (and is very safe) is to setup an ssh +key (see the ssh-keygen manpage) and ensure that ssh-agent forwarding is turned +on in your ssh client config (e.g. "ForwardAgent yes"). You would put +the public version of your key onto the middle and target systems (in the +~/.ssh/authorized_keys file), and the private key on your local system (which +I recommend you encrypt). With this setup, a series of ssh connections that +starts from the system where your private key is available will auto-authorize +(after a pass-phrase prompt on the first system if your key is encrypted). +See also ssh-agent for a way to keep a public key unlocked in memory for an +extended time, and the keychain project for a way to manage a system-wide, +per-user ssh-agent. + +

    To test that you have things setup right, first test a series of +remote-shell connections outside of rsync. A command such as the following +should work without issuing multiple prompts (use the appropriate remote-shell +and the substitute the real hostnames for "middle" and "target", of course): + +

    ssh middle ssh target uptime
    + +

    If you get a password/passphrase prompt to get into the first ("middle") +system that's fine, but the extra hop needs to occur without any extra user +interaction. + +

    Once that's done, you can do an rsync copy like this: + +

    rsync -av -e "ssh middle ssh" target:/src/ /dest/
    + +

    Method 2 -- requires ssh and nc (netcat)

    + +

    Assuming you're using ssh as your remote shell, you can configure ssh to +use a proxy command to get to the remote host you're interested in reaching. +Doing this will allow the multi-hop connection to work with rsync, even if +both hosts prompt for a password -- this is because both ssh connections +originate from the localhost, and thus both instances of ssh have access to +the local tty to use for an out-of-band password prompt. + +

    Here is an example config for your ~/.ssh/config file (substitute "target", +"target_user", and "middle" as appropriate): + +

    Host target
    +  ProxyCommand nohup ssh middle nc -w1 %h %p
    +  User target_user
    +
    + +

    This proxy setup uses "ssh" to login to the firewall system ("middle") and +uses "nc" (netcat) to connect to the target host ("target") using the default +port number. The use of "nohup" silences a warning at the end of the run, and +the "-w1" option tells nc to shut down when the connection closes. + +

    With this done, you can run a normal-looking rsync command to "target" that +will run the proxy command to get through the firewall system: + +

    rsync -av /src/ target:/dest/
    + +

    Method 3 -- an alternate ssh method for those without nc (netcat)

    + +

    Assuming you're using ssh as your remote shell, you can configure ssh to +forward a local port through your middle system to the ssh port (22) on the +target system. This method does not require the use of nc (it uses only +ssh to effect the extra hop), but otherwise it is similar to, but slightly +less convenient than, method 2. + +

    The first thing we need is an ssh configuration that will allow us to +connect to the forwarded port as if we were connecting to the target +system, and we need ssh to know what we're doing so that it doesn't +complain about the host keys being wrong. We can do this by adding this +section to your ~/.ssh/config file (substitute "target" and "target_user" +as appropriate, but leave "localhost" unchanged): + +

    Host target
    +  HostName localhost
    +  Port 2222
    +  HostKeyAlias target
    +  User target_user
    +
    + +

    Next, we need to enable the port forwarding: + +

    ssh -fN -l middle_user -L 2222:target:22 middle
    + +

    What this does is cause a connection to port 2222 on the local system to +get tunneled to the middle system and then turn into a connection to the +target system's port 22. The -N option tells ssh not to start a shell on +the remote system, which works with modern ssh versions (you can run a +sleep command if -N doesn't work). The -f option tells ssh to put the +command in the background after any password/passphrase prompts. + +

    With this done, you could run a normal-looking rsync command to "target" +that would use a connection to port 2222 on localhost automatically: + +

    rsync -av target:/src/ /dest/
    + +

    Note: starting an ssh tunnel allows anyone on the source system +to connect to the localhost port 2222, not just you, but they'd still need +to be able to login to the target system using their own credentials. + +

    Method 4 -- for using rsync in daemon-mode (requires nc)

    + +

    Install and configure an rsync daemon on the target and use ssh and nc +to send the socket data to the remote host. + +

    RSYNC_CONNECT_PROG='ssh -l middle_user middle nc %H 873' \
    +    rsync daemonuser@target::module/src/ /dest/
    + +

    (You can also export that variable into your environment if you want to +perform a series of daemon rsync commands through the same middle host.) + +

    This command takes advantage of the RSYNC_CONNECT_PROG environment +variable, which tells rsync to pipe its socket data to an external program +in place of making a direct socket connection. The command specifed above +uses ssh to run the nc (netcat) command on the middle host, which forwards +all socket data to port 873 on the target host (%H). The "%H" will be +substituted with the target host from the rsync command as long as you're +running rsync 3.0.0 or newer (you'll need to replace %H with the actual +target hostname for earlier rsync versions, which makes the hostname +specified in the rsync command superfluous). + +

    Method 5 -- for using rsync in daemon-mode (for those without nc)

    + +

    Install and configure an rsync daemon on the target and use an ssh +tunnel to reach the rsync sever. This is similar to method 3, but it +tunnels the daemon port for those that prefer to use an rsync daemon. + +

    (Note that this method also works to tunnel a socket connection +directly to a destination system if you use "localhost" as the target +hostname and your destination system's name as the middle hostname.) + +

    Installing the rsync daemon is beyond the scope of this document, but +see the rsyncd.conf manpage for more information. Keep in mind that you +don't need to be root to run an rsync daemon as long as you don't use a +protected port. + +

    Once your rsync daemon is up and running, you build an ssh tunnel +through your middle system like this: + +

    ssh -fN -l middle_user -L 8873:target:873 middle
    + +

    What this does is cause a connection to port 8873 on the local system to +turn into a connection from the middle system to the target system on port +873. (Port 873 is the normal port for an rsync daemon.) The -N option +tells ssh not to start a shell on the remote system, which works with +modern ssh versions (you can run a sleep command if -N doesn't work). The +-f option tells ssh to put the command in the background after any +password/passphrase prompts. + +

    Now when an rsync command is executed with a daemon-mode command-line +syntax to the local system, the conversation is directed to the target +system. For example: + +

    rsync -av --port 8873 localhost::module/src/ dest/
    +rsync -av rsync://localhost:8873/module/src/ dest/
    + +

    Note: starting an ssh tunnel allows anyone on the source system +to connect to the localhost port 8873, not just you, so you may want to +enable username/password restrictions on your rsync daemon. + + diff --git a/rsync-web/footer.html b/rsync-web/footer.html new file mode 100644 index 00000000..ba82ed3f --- /dev/null +++ b/rsync-web/footer.html @@ -0,0 +1,19 @@ + + + + +
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= + + +

    + + + + + + diff --git a/rsync-web/header.html b/rsync-web/header.html new file mode 100644 index 00000000..251c2bd5 --- /dev/null +++ b/rsync-web/header.html @@ -0,0 +1,40 @@ + + + + + + + + + + + +
    + + + rsync + + +
    + =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= +
    + + + +
    diff --git a/rsync-web/horus.txt b/rsync-web/horus.txt new file mode 100644 index 00000000..d861dc84 --- /dev/null +++ b/rsync-web/horus.txt @@ -0,0 +1,28 @@ +I do local backups on several of my machines using rsync. I have an +extra disk installed that can hold all the contents of the main +disk. I then have a nightly cron job that backs up the main disk to +the backup. This is the script I use on one of those machines. + + #!/bin/sh + + export PATH=/usr/local/bin:/usr/bin:/bin + + LIST="rootfs usr data data2" + + for d in $LIST; do + mount /backup/$d + rsync -ax --exclude fstab --delete /$d/ /backup/$d/ + umount /backup/$d + done + + DAY=`date "+%A"` + + rsync -a --delete /usr/local/apache /data2/backups/$DAY + rsync -a --delete /data/solid /data2/backups/$DAY + + + +The first part does the backup on the spare disk. The second part +backs up the critical parts to daily directories. I also backup the +critical parts using a rsync over ssh to a remote machine. + diff --git a/rsync-web/how-rsync-works.html b/rsync-web/how-rsync-works.html new file mode 100644 index 00000000..2a3fba9e --- /dev/null +++ b/rsync-web/how-rsync-works.html @@ -0,0 +1,350 @@ + + +How Rsync Works + + +

    How Rsync Works
    A Practical Overview

    +

    Foreword

    +

    +The original +Rsync technical report +and +Andrew Tridgell's +Phd thesis (pdf) +Are both excellent documents for understanding the +theoretical mathematics and some of the mechanics of the rsync algorithm. +Unfortunately they are more about the theory than the +implementation of the rsync utility (hereafter referred to as +Rsync). +

    +In this document I hope to describe... +

      +
    • A non-mathematical overview of the rsync algorithm. +
    • How that algorithm is implemented in the rsync utility. +
    • The protocol, in general terms, used by the rsync utility. +
    • The identifiable roles the rsync processes play. +
    +

    +This document be able to serve as a guide for programmers +needing something of an entré into the source code but the +primary purpose is to give the reader a foundation from +which he may understand +

      +
    • Why rsync behaves as it does. +
    • The limitations of rsync. +
    • Why a requested feature is unsuited to the code-base. +
    +

    +This document describes in general terms the construction +and behaviour of Rsync. In some cases details and exceptions +that would contribute to specific accuracy have +been sacrificed for the sake meeting the broader goals. +

    Processes and Roles

    +

    +When we talk about Rsync we use specific terms to refer to +various processes and their roles in the task performed by +the utility. For effective communication it is important that we +all be speaking the same language; likewise it is important +that we mean the same things when we use certain terms in a +given context. On the rsync mailing list there is often +some confusion with regards to role and processes. For +these reasons I will define a few terms +used in the role and process contexts that will be used henceforth. + + + + + + + + + +
    client +role + + The client initiates the synchronisation. +
    server +role + + The remote rsync process or system to which the + client connects either within a local transfer, via + a remote shell or via a network socket. +

    + This is a general term and should not be confused with the daemon. +

    + + + Once the connection between the client and server is established + the distinction between them is superseded by the + sender and receiver roles. +
    daemon +Role and process + + An Rsync process that awaits connections from + clients. On a certain platform this would be called a + service. +
    remote shell +role and set of processes + + One or more processes that provide connectivity + between an Rsync client and an Rsync server on a + remote system. +
    sender +role and process + + The Rsync process that has access to the source + files being synchronised. +
    receiver +role and process + + As a role the receiver is the destination system. + As a process the receiver is the process that + receives update data and writes it to disk. +
    generator +process + + The generator process identifies changed files and + manages the file level logic. +
    +

    +

    Process Startup

    +

    +When an Rsync client is started it will first establish a +connection with a server process. This connection may be +through pipes or over a network socket. +

    +When Rsync communicates with a remote non-daemon server via +a remote shell the startup method is to fork the remote +shell which will start an Rsync server on the remote system. +Both the Rsync client and server are communicating via pipes +through the remote shell. As far as the rsync processes are +concerned there is no network. +In this mode the rsync options for the server process are +passed on the command-line that is used to start the remote +shell. +

    +When Rsync is communicating with a daemon it is +communicating directly with a network socket. This is the +only sort of Rsync communication that could be called +network aware. +In this mode the rsync options must be sent over the socket, as +described below. +

    +At the very start of the communication between the client +and the server, they each send the maximum protocol version +they support to the other side. +Each side then uses the minimum value as the the protocol +level for the transfer. +If this is a daemon-mode connection, rsync options are sent +from the client to the server. Then, the exclude list is +transmitted. From this point onward the +client-server relationship is relevant only with regards +to error and log message delivery. +

    +Local Rsync jobs (when the source and destination are both on locally +mounted filesystems) are done exactly like a push. The +client, which becomes the sender, forks a server process to +fulfill the receiver role. The client/sender and +server/receiver communicate with each other over pipes. +

    The File List

    +The file list includes not only the pathnames but also +ownership, mode, permissions, size and modtime. +If the --checksum option has been specified it also includes +the file checksums. +

    +The first thing that happens once the startup has completed +is that the sender will create the file list. +While it is being built, each entry is transmitted to the +receiving side in a network-optimised way. +

    +When this is done, each side sorts the file list lexicographically by path +relative to the base directory of the transfer. +(The exact sorting algorithm varies depending on what protocol +version is in effect for the transfer.) +Once that has happened all references to files +will be done by their index in the file list. +

    +If necessary the sender follows the file list with id→name +tables for users and groups which the receiver will use to +do a id→name→id translation for every file in the file +list. +

    +After the file list has been received by the receiver, it +will fork to become the generator and receiver pair +completing the pipeline. +

    The Pipeline

    +Rsync is heavily pipelined. This means that it is a set of +processes that communicate in a (largely) unidirectional +way. Once the file list has been shared the pipeline +behaves like this: +
    + generator → sender → receiver +
    +

    +The output of the generator is input for the sender and the +output of the sender is input for the receiver. +Each process runs independently and is delayed only when the +pipelines stall or when waiting for disk I/O or CPU resources. +

    The Generator

    +

    +The generator process compares the file list with its local +directory tree. Prior to beginning its primary function, if +--delete has been specified, it will first identify local +files not on the sender and delete them on the receiver. +

    +The generator will then start walking the file list. Each +file will be checked to see if it can be skipped. In the +most common mode of operation files are not skipped if the +modification time or size differs. If --checksum was +specified a file-level checksum will be created and +compared. Directories, device nodes and symlinks are not +skipped. Missing directories will be created. +

    +If a file is not to be skipped, any existing version on the +receiving side becomes the "basis file" for the transfer, and is +used as a data source that will help to eliminate matching data +from having to be sent by the sender. To effect this remote +matching of data, block checksums are created for the basis file +and sent to the sender immediately following the file's index +number. +An empty block checksum set is sent for new files and if +--whole-file was specified. +

    +The block size and, in later versions, the size of the +block checksum are calculated on a per file basis according +to the size of that file. +

    The Sender

    +The sender process reads the file index numbers and associated +block checksum sets one at a time from the generator. +

    +For each file id the generator sends it will store the +block checksums and build a hash index of them for rapid lookup. +

    +Then the local file is read and a checksum is +generated for the block beginning with the first byte of the +local file. This block checksum is looked for in the +set that was sent by the generator, and if no match is found, +the non-matching byte will be appended to the non-matching data +and the block starting at the next byte will be compared. +This is what +is referred to as the “rolling checksum” +

    +If a block checksum match is found it is considered a +matching block and any accumulated non-matching data will be +sent to the receiver followed by the offset and length in +the receiver's file of the matching block and the block +checksum generator will be advanced to the next byte after +the matching block. +

    +Matching blocks can be identified in this way even if +the blocks are reordered or at different offsets. +This process is the very heart of the rsync algorithm. +

    +In this way, the sender will give the receiver instructions for +how to reconstruct the source file into a new destination file. +These instructions detail all the matching data that can be +copied from the basis file (if one exists for the transfe), +and includes any raw data that was not available locally. +At the end of each file's processing a whole-file +checksum is sent and the sender proceeds with the next +file. +

    +Generating the rolling checksums and searching for matches +in the checksum set sent by the generator require a good +deal of CPU power. Of all the rsync processes it is the +sender that is the most CPU intensive. +

    The Receiver

    +

    +The receiver will read from the sender data for each file +identified by the file index number. It will open the local +file (called the basis) and will create a temporary file. +

    +The receiver will expect to read non-matched data and/or to match +records all in sequence for the final file contents. When +non-matched data is read it will be written to the +temp-file. When a block match record is received the +receiver will seek to the block offset in the basis file +and copy the block to the temp-file. In this way the +temp-file is built from beginning to end. +

    +The file's checksum is generated as the temp-file is built. +At the end of the file, this checksum is compared with the +file checksum from the sender. If the file checksums do not +match the temp-file is deleted. If the file fails once it +will be reprocessed in a second phase, and if it fails twice +an error is reported. +

    +After the temp-file has been completed, its ownership and +permissions and modification time are set. It is then +renamed to replace the basis file. +

    +Copying data from the basis file to the temp-file make the +receiver the most disk intensive of all the rsync processes. +Small files may still be in disk cache mitigating this but +for large files the cache may thrash as the generator has +moved on to other files and there is further latency caused +by the sender. As +data is read possibly at random from one file and written to +another, if the working set is larger than the disk cache, +then what is called a seek storm can occur, further +hurting performance. +

    The Daemon

    +The daemon process, like many daemons, forks for every +connection. On startup, it parses the rsyncd.conf file +to determine what modules exist and to set the global options. +

    +When a connection is received for a defined module the +daemon forks a new child process to handle the connection. +That child process then reads the rsyncd.conf file to set +the options for the requested module, which may chroot to the +module path and may drop setuid and setgid for the +process. After that it will behave just like any other +rsync server process adopting either a sender or receiver +role. +

    The Rsync Protocol

    +

    +A well-designed communications protocol has a number of +characteristics. +

      +
    • Everything is sent in well defined packets with + a header and an optional body or data payload. +
    • In each packet's header a type and or command + specified. +
    • Each packet has a definite length. +
    +

    +In addition to these characteristics, protocols have varying degrees of +statefulness, inter-packet independence, human readability, +and the ability to reestablish a disconnected session. +

    +Rsync's protocol has none of these good characteristics. The data is +transferred as an unbroken stream of bytes. With the +exception of the unmatched file-data, there are no length +specifiers nor counts. Instead the meaning of each byte is +dependent on its context as defined by the protocol level. +

    +As an example, when the sender is sending the file list it +simply sends each file list entry and terminates the list +with a null byte. Within the file list entries, a bitfield +indicates which fields of the structure to expect and those +that are variable length strings are simply null terminated. +The generator sending file numbers and block checksum sets +works the same way. +

    +This method of communication works quite well on reliable +connections and it certainly has less data overhead than the +formal protocols. It unfortunately makes the protocol +extremely difficult to document, debug or extend. +Each version of the protocol will have subtle differences on +the wire that can only be anticipated by knowing the exact +protocol version. +

    notes

    +This document is a work in progress. The author expects +that it has some glaring oversights and some portions that may be +more confusing than enlightening for some readers. It is +hoped that this could evolve into a useful reference. +

    +Specific suggestions for improvement are welcome, as would be a +complete rewrite. + + diff --git a/rsync-web/index.html b/rsync-web/index.html new file mode 100644 index 00000000..511f365f --- /dev/null +++ b/rsync-web/index.html @@ -0,0 +1,528 @@ + + + +rsync + + + + +

    Welcome to the rsync web pages

    + +rsync is an open source +utility that provides fast incremental file transfer. rsync is freely +available under the GNU General Public +License and is currently being maintained by +Andrew Tridgell. + +

    A full changelog of all the releases, including upcoming releases, is in the +NEWS file. + +

    + +

    Rsync version 3.4.3 released

    +May 20th, 2026 + +

    Rsync version 3.4.3 has been released. This is a major security release. + +

    See the 3.4.3 NEWS for a detailed changelog. +The latest manpages are also available for:

    + +

    The source tar is available here: +rsync-3.4.3.tar.gz +(signature), +and the diffs from version 3.4.2 are available here: +rsync-3.4.2-3.4.3.diffs.gz +(signature). + +

    Rsync version 3.4.2 released

    +April 28th, 2026 + +

    Rsync version 3.4.2 has been released. + +

    See the 3.4.2 NEWS for a detailed changelog. +The latest manpages are also available for:

    + +

    The source tar is available here: +rsync-3.4.2.tar.gz +(signature), +and the diffs from version 3.4.1 are available here: +rsync-3.4.1-3.4.2.diffs.gz +(signature). + +

    Rsync version 3.4.1 released

    +January 15th, 2025 + +

    Rsync version 3.4.1 has been released. +This is a fix for some regressions in 3.4.0 + +

    See the 3.4.1 NEWS for a detailed changelog. +The latest manpages are also available for:

    + +

    The source tar is available here: +rsync-3.4.1.tar.gz +(signature), +with a tar file of the "rsync-patches" repository released in a separate file: +rsync-patches-3.4.1.tar.gz +(signature), +and the diffs from version 3.4.0 are available here: +rsync-3.4.0-3.4.1.diffs.gz +(signature). + + +

    Rsync version 3.4.0 released

    +January 14th, 2025 + +

    Rsync version 3.4.0 has been released. +This is a security release, fixing several important security vulnerabilities. + +

    See the 3.4.0 NEWS for a detailed changelog. + +

    The source tar is available here: +rsync-3.4.0.tar.gz +(signature), +with a tar file of the "rsync-patches" repository released in a separate file: +rsync-patches-3.4.0.tar.gz +(signature), +and the diffs from version 3.3.0 are available here: +rsync-3.3.0-3.4.0.diffs.gz +(signature). + +

    Rsync version 3.3.0 released

    +April 6th, 2024 + +

    Rsync version 3.3.0 has been released. +This is a bug-fix release. + +

    See the 3.3.0 NEWS for a detailed changelog. + +

    The source tar is available here: +rsync-3.3.0.tar.gz +(signature), +with a tar file of the "rsync-patches" repository released in a separate file: +rsync-patches-3.3.0.tar.gz +(signature), +and the diffs from version 3.2.2 are available here: +rsync-3.2.7-3.3.0.diffs.gz +(signature). + +


    +

    Rsync version 3.2.7 released

    +October 20th, 2022 + +

    Rsync version 3.2.7 has been released. +This has some new features & fixes, including various bug fixes for arg validation & filter-rule validation. + +

    See the 3.2.7 NEWS for a detailed changelog. + +

    The source tar is available here: +rsync-3.2.7.tar.gz +(signature), +with a tar file of the "rsync-patches" repository released in a separate file: +rsync-patches-3.2.7.tar.gz +(signature), +and the diffs from version 3.2.2 are available here: +rsync-3.2.6-3.2.7.diffs.gz +(signature). + +


    +

    Rsync version 3.2.6 released

    +September 9th, 2022 + +

    Rsync version 3.2.6 has been released. +This is a bug-fix release. + +

    See the 3.2.6 NEWS for a detailed changelog. + +

    The source tar is available here: +rsync-3.2.6.tar.gz +(signature), +with a tar file of the "rsync-patches" repository released in a separate file: +rsync-patches-3.2.6.tar.gz +(signature), +and the diffs from version 3.2.2 are available here: +rsync-3.2.5-3.2.6.diffs.gz +(signature). + +


    +

    Rsync version 3.2.5 released

    +August 14th, 2022 + +

    Rsync version 3.2.5 has been released. +This is a bug-fix and security release. + +

    See the 3.2.5 NEWS for a detailed changelog. + +


    +

    Rsync version 3.2.4 released

    +April 15th, 2022 + +

    Rsync version 3.2.4 has been released. +Another typical release with both bug fixes and some enhancements. It also contains a +security fix +for the bundled zlib 1.2.8, which may or may not be used in your particular build configuration. + +

    See the 3.2.4 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.2.3 released

    +August 6th, 2020 + +

    Rsync version 3.2.3 has been released. +It contains a smattering of bug fixes and various enhancements. + +

    See the 3.2.3 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.2.2 released

    +July 4th, 2020 + +

    Rsync version 3.2.2 has been released. +This is a few more portability fixes, some improvements to the newest features, and some +other simple changes. Hopefully this will be the last of these recent touch-up releases. + +

    See the 3.2.2 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.2.1 released

    +June 22th, 2020 + +

    Rsync version 3.2.1 has been released. +This is mainly a few fixes for some release issues and portability problems. +There's also a couple new features, just for good measure. + +

    See the 3.2.1 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.2.0 released

    +June 19th, 2020 + +

    Rsync version 3.2.0 has been released. +This release has a good number of a few new features and various bug fixes. + +

    See the 3.2.0 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.1.3 released

    +January 28th, 2018 + +

    Rsync version 3.1.3 has been released. +This release has a +couple security fixes, +a few new features, and various bug fixes. + +

    See the 3.1.3 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.1.2 released

    +December 21st, 2015 + +

    Rsync version 3.1.2 has been released. This is a bug-fix release. +It includes a security fix +for a transfer from a sender that you don't fully trust. + +

    See the 3.1.2 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.1.1 released

    +June 22nd, 2014 + +

    Rsync version 3.1.1 has been released. This is a bug-fix release. + +

    See the 3.1.1 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.1.0 released

    +September 28th, 2013 + +

    Rsync version 3.1.0 has been released. This is a +feature release that improves performance, provides several new options, and +fixes a few bugs along the way. + +

    See the 3.1.0 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.0.9 released

    +September 23th, 2011 + +

    Rsync version 3.0.9 has been released. This is a bug-fix release. + +

    See the 3.0.9 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.0.8 released

    +March 26th, 2011 + +

    Rsync version 3.0.8 has been released. This is a bug-fix release. + +

    See the 3.0.8 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.0.7 released

    +December 31th, 2009 + +

    Rsync version 3.0.7 has been released. This is a bug-fix release. + +

    See the 3.0.7 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.0.6 released

    +May 8th, 2009 + +

    Rsync version 3.0.6 has been released. This is a bug-fix release. + +

    See the 3.0.6 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.0.5 released

    +December 28th, 2008 + +

    Rsync version 3.0.5 has been released. This is another bug-fix release. + +

    See the 3.0.5 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.0.4 released

    +September 6th, 2008 + +

    Rsync version 3.0.4 has been released. This is a bug-fix release with the +only enhancement being the adding of a way to interact with an +overly-restrictive server that refuses rsync's behind-the-scenes use of the -e +option. + +

    See the 3.0.4 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.0.3 released

    +June 29th, 2008 + +

    Rsync version 3.0.3 has been released. This is a bug-fix release that has +no new features (though it does have one new script in the support directory). + +

    See the 3.0.3 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.0.2 released

    +April 8th, 2008 + +

    Rsync version 3.0.2 has been released. This is a +security release +that fixes a potential buffer-overflow issue. + +

    See the 3.0.2 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.0.1 released

    +April 3rd, 2008 + +

    Rsync version 3.0.1 has been released. This is a bug-fix release, which also +includes fixes/improvements for several issues in the daemon-exclude code. + +

    See the 3.0.1 NEWS +for a detailed changelog. + +


    +

    Rsync version 3.0.0 released

    +March 1st, 2008 + +

    Rsync version 3.0.0 is finally here! This is a feature release that +also includes quite a few bug fixes. + +

    The 3.0.0 version number is such a large bump up from 2.6.9 due to the +addition of an +incremental recursion scan (which helps a lot with large transfers) and the +official arrival of several other new features, including ACL support, extended +attribute support, filename character-set conversion, etc. + +

    See the 3.0.0 NEWS +for a detailed changelog. + +


    +

    Rsync version 2.6.9 released

    +November 6th, 2006 + +

    Rsync version 2.6.9 has been released. This is primarily a bug-fix +release with a few minor new features. + +

    See the 2.6.9 NEWS +for a detailed changelog. + +


    +

    Rsync version 2.6.8 released

    +April 22th, 2006 + +

    Rsync version 2.6.8 has been released. This is a bug-fix release that +primarily addresses an exclude problem that affected the --relative option, +but also includes a security fix for +the xattrs.diff patch (which is not an +official part of rsync, but some packagers include it in their release). + +

    See the 2.6.8 NEWS +for a detailed changelog. + +


    +

    Rsync 2.6.7 released

    +March 11th, 2006 + +

    Rsync version 2.6.7 has been released. This release has both several new +features and the usual accompaniment of bug fixes. + +

    See the 2.6.7 NEWS +for a detailed changelog. + +


    +

    Rsync 2.6.6 released

    +July 28th, 2005 + +

    Rsync version 2.6.6 has been released. This release is a bug-fix release +which contains a security fix +to handle a null-pointer bug that turned up in rsync's version of zlib +1.1.4 (this is not the recent zlib 1.2.2 security fix, which did not +affect rsync) and to squash a few other minor bugs. To deal with the +zlib issue, rsync has been upgraded to include zlib 1.2.3. + +

    See the 2.6.6 NEWS +for a detailed changelog. + +


    +

    Rsync 2.6.5 released

    +June 1st, 2005 + +

    Rsync version 2.6.5 has been released. This release is primarily a bug-fix +release to squash some annoying problems that made it into the (feature-filled) +release of 2.6.4, plus a few minor enhancements. + +

    See the 2.6.5 NEWS +for a detailed changelog. + +


    +

    Rsync 2.6.4 released

    +March 30th, 2005 + +

    Rsync version 2.6.4 has been released. This release combines quite a +few new features, some improved delete efficiency, and the usual array of +bug fixes. + +

    See the 2.6.4 NEWS +for a detailed changelog. + +


    +

    Rsync 2.6.3 released

    +September 30th, 2004 + +

    Rsync version 2.6.3 has been released. It contains several new features +and quite a few bug fixes, including a security +fix for a patch-sanitizing bug in the daemon code. + +

    See the 2.6.3 NEWS +for a detailed changelog. + +


    +

    Rsync 2.6.2 released

    +April 30th, 2004 + +

    Rsync version 2.6.2 has been released. It is a bug-fix release that mainly +fixes a bug with the --relative option (-R) in 2.6.1 +that could cause files to be transferred incorrectly. This only affected a +source right at the root of the filesystem, such as "/" or "/*" (if you +first "cd /" and then copy from ".", it would not tickle the bug). + +

    See the 2.6.2 NEWS +for a detailed changelog. + +


    +

    Rsync 2.6.1 released

    +April 26th, 2004 + +

    Rsync version 2.6.1 has been released. It is primarily a performance +release that requires less memory to run, makes fewer write calls to the socket +(lowering the system CPU time), does less string copying (lowering the user CPU +time), and also reduces the amount of data that is transmitted over the wire. +There have also been quite a few bug fixes, including a +security fix for a daemon problem when chroot +is not enabled. See the +2.6.1 NEWS +for a detailed changelog. + +


    +

    One Cygwin hang-problem resolved

    + +

    The problem with rsync hanging at the end of the transfer on +Cygwin had been previously traced to a +signal-handling bug in their compatibility DLL. This bug appears to now be +fixed in DLL version 1.5.7-1, and Cygwin users are reporting that upgrading the +DLL removes the hang-at-end-of-transfer problem for their existing rsync executable. +(Note that this doesn't solve a hang that some folks see in the middle of a +transfer -- using daemon mode instead of ssh can work around that one.) + +


    +

    Rsync 2.6.0 released

    +January 1st, 2004 + +

    Two important things to note in the new release: + +

      + +
    1. The default remote shell is now "ssh" unless you tell configure you want to +make something else the default. + +
    2. Some bug fixes in the include/exclude code, while making things work +properly, have resulted in some user-visible changes for certain wildcard +strings. Read the BUG FIXES section in the +2.6.0 NEWS +to see if any of these changes apply to you. +(Most people should be unaffected.) + +
    + +

    One other item of note is that the oft-requested option "--files-from" is now +available. This option lets you specify a list of files to transfer, and can +be much more efficient than a recursive descent using include/exclude +statements (if you know in advance what files you want to transfer). The list +of files can come from either side of the connection, so it is possible for a +server to provide the file-list that lets someone grab a server-specified set of +files, for example. See the rsync man page +for more details. + +

    See the 2.6.0 NEWS +for a detailed changelog. + + diff --git a/rsync-web/issues.html b/rsync-web/issues.html new file mode 100644 index 00000000..dac9dd22 --- /dev/null +++ b/rsync-web/issues.html @@ -0,0 +1,210 @@ + + + +rsync current issues + + + + +

    current issues and debugging

    + +
      + +
    1. Q: + +Rsync appears hung -- what should I do? + +

      A: + +When experiencing a hang or freeze please gather the following +information before killing the rsync process: + +

        + +
      • The state of the send/receive queues shown with netstat on the two ends. + +
      • The system call that each of the 3 processes is stuck in (use truss on +solaris, strace on Linux, etc.). + +
      + +

      Try telling rsync on both sides of the connection to send messages to +stderr, which might make the failure message visible. i.e., use: + + +--msgs2stderr -M--msgs2stderr + + +

      That alone might get rsync to stop hanging. Also, if you're using more than +one --verbose (-v) option then I have 2 simple words +for you: stop it. If you need more info on what rsync is changing, using the +--itemize-changes option (-i) and repeat it if you +need to see unchanged files. This is a much better way to go that doesn't fill +up the communication pipeline with a large quanity of debug messages. + +

      See the "rsync-debug" script below for an example of how to grab strace +information from the remote rsync process(es). If you need help, send email to +the mailing list. + +

    2. Q: + +Why does my chrooted rsync daemon crash when doing an LDAP lookup for a user or +group? + +

      A: + +There is a bug in some LDAP libraries (e.g. Fedora Core 3) where it crashes +when someone looks up a name from inside a chrooted process (one that does not +contain copies of the libraries to perform the lookup). This is a bug that the +LDAP libraries will need to fix, and is out of rsync's hands. You can work +around the problem by using the --numeric-ids option, turning +chroot off, or getting rid of LDAP lookups. + +

    3. Q: + +Why does my transfer die with something like the following error? + +

      
      +rsync: error writing 4 unbuffered bytes - exiting: Broken pipe
      +rsync error: error in rsync protocol data stream (code 12) at io.c(463)
      +
      + +

      or + +

      
      +rsync: connection unexpectedly closed (24 bytes read so far)
      +rsync error: error in rsync protocol data stream (code 12) at io.c(342)
      +
      + +

      A: + +This error tells you that the local rsync was trying to talk to the remote +rsync, but the connection to that rsync is now gone. The thing you must +figure out is why, and that can involve some investigative work. + +

      It is a good idea use the --msgs2stderr options mentioned at +the top of this page to get rsync to output any errors it encounters to stderr +instead of trying to write them down the failing pipeline. + +

      If the connection is via ssh (or other remote-shell command) then you should +run some tests to make sure that you can actually run the remote rsync and that +your shell isn't injecting extraneous output into the rsync stream. For instance, +try running these two commands using whatever HOST (and user) options you need: + +

      
      +echo hi | ssh HOST cat
      +ssh HOST rsync --version
      +
      + +

      The first command should output just the string "hi" and nothing else. The +second command should successfully start the remote rsync and report its version. + +

      If the remote rsync is a daemon, your first step should be to look at the +daemon's log file to see if it logged an error explaining why it aborted the +transfer. Also double-check to ensure that the log file is setup right, as a +wrong "log file" setting in your rsyncd.conf file can also cause this problem. +You could also halt the daemon and run it interactively using the +--no-detach and --msgs2stderr options and look for +errors while someone tries the rsync copy in another window. + +

      As for the cause of the remote rsync going away, there are several +common issues that people run into: + +

        + +
      • The destination disk is full (remember that you need at least the +size of the largest file that needs to be updated available in free +disk space for the transfer to succeed). + +
      • An idle connection caused a router or remote-shell server to close +the connection. + +
      • A network error caused the connection to be dropped. + +
      • The remote rsync executable wasn't found. + +
      • Your remote-shell setup isn't working right or isn't "clean" +(i.e. it is sending spurious text to rsync). + +
      + +

      If you think the problem might be an idle connection getting closed, you +might be able to work around the problem by using a --timeout +option (newer rsyncs send keep-alive messages during lulls). You can also +configure ssh to send keep-alive messages when using Protocol 2 (look for +KeepAlive, ServerAliveInterval, ClientAliveInterval, ServerAliveCountMax, and +ClientAliveCountMax). You can also avoid some lulls by switching from +--delete (aka --delete-before) to --del +(aka --delete-during). + +

      If you can't figure out why the failure happened, there are steps +you can take to debug the situation. One way is to create a shell +script on the remote system such as +this one named "rsync-debug". +You would use the script like this: + +

      
      +rsync -av --rsync-path=/some/path/rsync-debug HOST:SOURCE DEST
      +rsync -av --rsync-path=/some/path/rsync-debug SOURCE HOST:DEST
      +
      + +

      This script enables core dumps and also logs all the OS system calls +that lead up to the failure to a file in the /tmp dir. You can use the +resulting files to help figure out why the remote rsync failed. + +

      If you are rsyncing directly to an rsync daemon (without using a +remote-shell transport), the above script won't have +any effect. Instead, halt the current daemon and run a debug version +with core-dumps enabled and (if desired) using a +system-call tracing utility such as strace, truss, or +tusc. For strace, you would do it like this (the -f option +tells strace to follow the child processes too): + +

      
      +ulimit -c unlimited
      +strace -f -t -s 1024 -o /tmp/rsync-$$.out rsync --daemon --no-detach
      +
      + +

      Then, use a separate window to actually run the failing transfer, after +which you can kill the debug rsync daemon (pressing Ctrl-C should do it). + +

      If you are using rsync under inetd, I'd suggest temporarily disabling +that and using the above daemon approach to debug what is going on. + +

    4. Q: + +Why does my connection to an rsync daemon (using the "::" syntax) +fail immediately with an error like the following? + +

      
      +rsync: connection unexpectedly closed (24 bytes read so far)
      +rsync error: error in rsync protocol data stream (code 12) at io.c(342)
      +
      + +

      A: + +Older rsync daemons (before 2.6.3) were unable to return errors that were +generated during the option-parsing phase of the transfer. Look in the +logfile on the server to see if an error was reported, such as a "refused" +option, an option that the server rsync doesn't support (e.g. perhaps +links are not supported by the server), or some other failure (such as +trying to send data to a read-only module). Upgrading the version of rsync +that is running as a daemon to at least 2.6.3 will allow these errors to +get returned to all rsync clients, old or new alike. + +

    + + diff --git a/rsync-web/lists.html b/rsync-web/lists.html new file mode 100644 index 00000000..8d0f1739 --- /dev/null +++ b/rsync-web/lists.html @@ -0,0 +1,66 @@ + + + +rsync mailing lists + + + +

    rsync mailing lists

    + +

    + There are three mailing lists for rsync. All are optionally + available in digest mode and also through web archives. +

    + +

    + Since these lists can generate a lot of traffic we suggest that + you should not subscribe from a web-mail account such as Yahoo! + or Hotmail, because your mailbox is likely to overflow. + Instead, please read directly from the archives. +

    + +
      +
    • + rsync is the main mailing lists for developers and + users. It sees between zero and twenty messages per day. + This is a good place to send questions about rsync, but please + read + this before posting. + + [archive 1, + archive 2, + subscriptions] +
    • + + +
    • + rsync-announce carries only messages from the + maintainer annoucing new releases, which happen at most a few + times per month. + + [archive 1, + archive 2, + subscriptions] +
    • + + +
    • + rsync-cvs carries messages are automatically generated + whenever a developer changes the source code, which can happen + many times per day. If you subscribe, you should probably + choose digest mode. + + [archive 1, + archive 2, + subscriptions] +
    • +
    + +

    + Please report problems with the lists to the postmaster + at samba.org, but note that you can control your own + subscription using the web interface. +

    + + diff --git a/rsync-web/newrsynclogo.jpg b/rsync-web/newrsynclogo.jpg new file mode 100644 index 00000000..70e7cbac Binary files /dev/null and b/rsync-web/newrsynclogo.jpg differ diff --git a/rsync-web/newrsynclogo.xcf b/rsync-web/newrsynclogo.xcf new file mode 100644 index 00000000..afaf6485 Binary files /dev/null and b/rsync-web/newrsynclogo.xcf differ diff --git a/rsync-web/nt.html b/rsync-web/nt.html new file mode 100644 index 00000000..cfbd1be1 --- /dev/null +++ b/rsync-web/nt.html @@ -0,0 +1,85 @@ + + + +rsync on NT + + + +

    rsync on NT

    + +
    +From:   "Mike McHenry" 
    +Subject: Rsync 2.3.1 WinNT binaries and instructions available
    +Date:   Fri, 15 Oct 1999 02:53:30 +1000
    +
    +Hello all,
    +
    +I have created Windows NT binaries for rsync 2.3.1 and have decided to make
    +them available for others to download. These binaries have been tested on
    +Windows NT Server 4.0 SP5 and WILL run in daemon mode if you follow my
    +instructions below. I make no guarantees about these binaries, they have
    +however been working for me for weeks on several NT machines.
    +
    +Binaries at ftp://ftp.minn.net/usr/mmchen/
    +
    +Instructions for running in daemon mode:
    +
    +1. You will need two files, rsync.exe and cygwin1.dll. Place rsync.exe
    +anywhere you like (I chose c:\program files\rsync\rsync.exe) and put
    +cygwin1.dll in c:\winnt\system32
    +
    +2. You will need a program from the NT Server resource kit called
    +srvany.exe. This program allows you to run any executable as a service. If
    +you simply install the entire service pack it will be located in c:\ntreskit
    +
    +3. Create a service for rsync by typing the following:
    +  instsrv Rsync "C:\ntreskit\srvany.exe"
    +
    +4. You should now have a new service called Rsync and you can verify by
    +looking in Start->Control Panel->Services DON'T START IT YET!
    +
    +5. If you want to run rsync in daemon mode you will need a configuration
    +file. Here is the one I use, call it rsyncd.conf and place it in the same
    +directory as rsync (C:\Program files\rsync\rsyncd.conf)
    +  use chroot = false
    +  strict modes = false
    +  hosts allow = *
    +
    +  [backup]
    +     path = /
    +     read only = yes
    +     list = no
    +
    +This example configuration will make one big anonymous anonymous rsync area
    +available, I use this to backup my NT machines from a central Unix machine.
    +This configuration might not be ideal for you, change to suit your tastes.
    +The first two lines are important for rsync to work on Windows NT however.
    +
    +6. You are going to need to hack some keys in the registry to make it work.
    +Don't do this unless you are comfortable with the changes! Run regedit32 and
    +add the following keys and values (quotation marks ARE IMPORTANT):
    +  HKEY_LOCAL_MACHINE->SYSTEM->CurrentControlSet->Services->Rsync
    +    Edit->Add Key->  Key Name: Paramaters
    +    Edit->Add Value->  Value Name: AppDirectory  Value: "C:\programfiles\rsync"
    +    Edit->Add Value->  Value Name: Application  Value: "C:\programfiles\rsync\rsync.exe"
    +    Edit->Add Value->  Value Name: AppParamters  Value: --config="C:\programfiles\rsync\rsyncd.conf" --daemon
    +
    +7. That's it, you should be able to start and stop the rsync service at will
    +using the Services Control Panel. When running with the above configuration
    +you should be able to test by attempting to telnet to port 873 from a remote
    +machine.
    +  telnet rsync.server.com 873 (replacing rsync.server.com with your own
    +server's address)
    +You should get a connection to the rsync daemon running on your NT box.
    +
    +8. If you have problems you are on your own, sorry, I have enough to do :) I
    +would suggest triple-checking your spelling on EVERYTHING (filenames,
    +configs, reg keys). If you have any comments or suggestions I would be happy
    +to hear them at mmchen@minn.net.
    +
    +Mike McHenry
    + Systems Administrator
    + MinnNet Communications, Inc.
    +
    + + diff --git a/rsync-web/nt.txt b/rsync-web/nt.txt new file mode 100644 index 00000000..a2a4311e --- /dev/null +++ b/rsync-web/nt.txt @@ -0,0 +1,74 @@ +From: "Mike McHenry" +Subject: Rsync 2.3.1 WinNT binaries and instructions available +Date: Fri, 15 Oct 1999 02:53:30 +1000 + + +Hello all, + +I have created Windows NT binaries for rsync 2.3.1 and have decided to make +them available for others to download. These binaries have been tested on +Windows NT Server 4.0 SP5 and WILL run in daemon mode if you follow my +instructions below. I make no guarantees about these binaries, they have +however been working for me for weeks on several NT machines. + +Binaries at ftp://ftp.minn.net/usr/mmchen/ + +Instructions for running in daemon mode: + +1. You will need two files, rsync.exe and cygwin1.dll. Place rsync.exe +anywhere you like (I chose c:\program files\rsync\rsync.exe) and put +cygwin1.dll in c:\winnt\system32 + +2. You will need a program from the NT Server resource kit called +srvany.exe. This program allows you to run any executable as a service. If +you simply install the entire service pack it will be located in c:\ntreskit + +3. Create a service for rsync by typing the following: + instsrv Rsync "C:\ntreskit\srvany.exe" + +4. You should now have a new service called Rsync and you can verify by +looking in Start->Control Panel->Services DON'T START IT YET! + +5. If you want to run rsync in daemon mode you will need a configuration +file. Here is the one I use, call it rsyncd.conf and place it in the same +directory as rsync (C:\Program files\rsync\rsyncd.conf) + use chroot = false + strict modes = false + hosts allow = * + + [backup] + path = / + read only = yes + list = no + +This example configuration will make one big anonymous anonymous rsync area +available, I use this to backup my NT machines from a central Unix machine. +This configuration might not be ideal for you, change to suit your tastes. +The first two lines are important for rsync to work on Windows NT however. + +6. You are going to need to hack some keys in the registry to make it work. +Don't do this unless you are comfortable with the changes! Run regedit32 and +add the following keys and values (quotation marks ARE IMPORTANT): + HKEY_LOCAL_MACHINE->SYSTEM->CurrentControlSet->Services->Rsync + Edit->Add Key-> Key Name: Paramaters + Edit->Add Value-> Value Name: AppDirectory Value: "C:\programfiles\rsync" + Edit->Add Value-> Value Name: Application Value: "C:\programfiles\rsync\rsync.exe" + Edit->Add Value-> Value Name: AppParamters Value: --config="C:\programfiles\rsync\rsyncd.conf" --daemon + +7. That's it, you should be able to start and stop the rsync service at will +using the Services Control Panel. When running with the above configuration +you should be able to test by attempting to telnet to port 873 from a remote +machine. + telnet rsync.server.com 873 (replacing rsync.server.com with your own +server's address) +You should get a connection to the rsync daemon running on your NT box. + +8. If you have problems you are on your own, sorry, I have enough to do :) I +would suggest triple-checking your spelling on EVERYTHING (filenames, +configs, reg keys). If you have any comments or suggestions I would be happy +to hear them at mmchen@minn.net. + +Mike McHenry + Systems Administrator + MinnNet Communications, Inc. + diff --git a/rsync-web/resources.html b/rsync-web/resources.html new file mode 100644 index 00000000..6b6d11ad --- /dev/null +++ b/rsync-web/resources.html @@ -0,0 +1,127 @@ + + + +rsync resources + + + +

    rsync resources

    + +Please let us know if you have any rsync-related +documents to add to this list: + +
      + +
    • Be sure to search for the latest rsync info to get up-to-the-minute +results. You can use the search box at the top of the page for either +web searching or project searching (they are done via Google). + +
    • +2002-05-15: rsync is not official GNU software, but we try to +work more or less in accordance with their Guidelines for +Maintaining GNU Software. + +
    • 2002-04-10: A new tutorial on using rsync to create a system of rotating +backups, by Mike Rubel. + +
    • If you still don't know what rsync is, then take a look at the +README. + +
    • There is now a python script that implements +an +atomic update of the received files at the end of the transfer (when pulling). + +
    • Brian Elliott Finley has put together a great Linux install system based +on rsync. You you read about it at http://thefinleys.com/SystemImager/ + +
    • Dirvish is a fast, disk based, +rotating network backup system that was originally written by JW Schultz. + +
    • BackupPC: a backup +system using rsync. Hard-links all identical files (even between multiple +runs and multiple backup sources), compresses the files, provides an easy +interface to find and restore files, etc. + +
    • Bung: (BackUp Next +Generation) backs up files, MariaDB, OpenLDAP, postgres, etc. via an extensible +templates system with git support. The rsync-based "rolling full" backup is +easy to browse and restore from using everyday tools. + +
    • drsync: a wrapper for rsync +that remembers file sets between invocations so that a 2-way synchronization +of two systems is possible. + +
    • rsyncbackup: a helper +script that uses config files to setup multiple backup scenarios and +invokes rsync (or rsyncX on macOS). + +
    • Users who use the new character-set conversion option of rsync (--iconv) +may want to check into the convmv +package that lets you convert the names of already-transferred files into a +new characterset (for when you want to change or normalize the characterset +of a hierarchy of files). + +
    • For those wanting to use launchd to run an rsync daemon (e.g. Mac +OS X Tiger users), Glen Scott provides the necessary +rsync.plist +file. + +
    • For the developer wanting to work on a branched rsync version based on +one of the diffs in the patches dir, you may want to check into Matt's +patchsync +script. + + + +
    • There are a few choices for making rsync work with OS X's resource forks. +One is the official apple patch found on their opendarwin site, such as +this one +(I've heard patch inefficiently transfers the entire resource fork information +for every file on every transfer.) Another choice is to use a third-party +adapted rsync, such as +rsyncx or a +rsync+hfsmode +patch by D Andrew Reynhout. For the future, I would like to see an rsync +that supports ACLs and Posix xattrs adapted to interact with resource forks in +a seamless way (if that's possible). + +
    • Piero Orsoni wrote a GTK-based GUI for rsync called +grsync. + +
    • Those interested in using an rsync daemon over SSL may be interested in +this wiki page +that outlines a way to use a modern, simplified stunnel setup. + +
    • Thomas Roessler has written an rsync wrapper for +efficient, +safe CVS mirroring. + +
    • Rsync is distributed with the +rrsync python script +that lets you restrict the rsync commands that can be run via ssh. (This is +a enhanced and reworked version of Joe Smith's original perl version.) + +
    • Lee Eakin has written a perl wrapper for rsync. + +
    • A wire-compatible rsync implementation in perl. + +
    • A REXX implementation of rsync. + +
    • An initial version of a rewrite of rsync for .Net. + +
    • You might want to check out an encryption program that is being developed +to produce more rsync-friendly output: +rsyncrypto. + +
    • If you need a 2-way synchronization because both ends of the transfer may +be changing files, you may want to either look into a tool designed to do this +(e.g. unison), or you may +wish to use an external wrapper for rsync that keeps extra data about what was +in the last transfer so that it can figure out if a file is new or deleted +(e.g. drsync). + +
    + + diff --git a/rsync-web/rsync-and-debian/.cvsignore b/rsync-web/rsync-and-debian/.cvsignore new file mode 100644 index 00000000..a426175c --- /dev/null +++ b/rsync-web/rsync-and-debian/.cvsignore @@ -0,0 +1,2 @@ +rsync-and-debian.html +rsync-and-debian.ps diff --git a/rsync-web/rsync-and-debian/rsync-and-debian.sgml b/rsync-web/rsync-and-debian/rsync-and-debian.sgml new file mode 100644 index 00000000..ab9faaf4 --- /dev/null +++ b/rsync-web/rsync-and-debian/rsync-and-debian.sgml @@ -0,0 +1,869 @@ + + +
    + + About integration of rsync and Debian + + Martin Pool mbp@samba.org + + $Date: 2002/05/14 03:10:09 $ + + + + + + Introduction + +

    + It seems like there is an rsync thread on + +

    + By way of background, I have been the maintainer of rsync for + about a year, and I wrote the + +

    + I have tried to respond to all the issues raised on the list + recently, but possibly have missed some. After putting many + hours into rsync I suppose I am quite fond of it and may be a + little biased, but I think I also know its strengths and + problems more than most. + +

    + If there are issues ommitted by this document, or if you think + the answers are incomplete, unbalanced, or incorrect, then + please mail + + + + Background + + + + The rsync algorithm + +

    + rsync is really two independent but related ideas: the + + +

    + The rsync algorithm is described in detail in the ANU + Technical Report included with the distribution and on the + web site. +

    + +

    + Briefly, the rsync algorithm provides an efficient means to + transfer a file + +

    + The algorithm could be embodied in other forms and uses than + the rsync program. Indeed, the algorithm has been + reimplemented in programs other than rsync itself: in other + languages (Rexx, CAML, Java), in a C library ( + +

    + rsync deltas are much more efficient than diffs, for several + reasons: most importantly, useful diff formats include + context lines, which are wasted information. rsync gets the + same benefit of making sure that the change is applied to a + compatible version of the file, but in a much more efficient + way. diffs suffer from being human-readable and therefore + verbose, and cannot handle binary files. Both can be + transparently compressed. diffs are an invaluable tool for + software development, but not ideal for distribution of + deltas. +

    + + + + + + + The rsync mirroring program is similar in functionality to + other tools such as + +

    + In general, rsync is substantially faster than other + tools, because it uses delta compression, and because the + protocol is designed to be efficient both in traffic and + round trips. rsync can use + +

    + rsync has the important property that it is generally + idempotent: repeated runs of rsync, even if interrupted, + make the contents of the destination machine converge + towards those of the source machine. + +

    + Unlike wget and ftpmirror, rsync must be installed on both + the client and the server. It uses its own protocol, rather + than HTTP or FTP. This is probably the biggest practical + drawback at the moment: one cannot use rsync from arbitrary + web sites, but only from systems that have specifically + chosen to support it. On the other hand, many important + sites for free software do now support rsync, and the + developers see them as an important constituency to support. + +

    + rsync can tunnel through a proxy server that supports + + rsync can run as a daemon, similar to an /etc/rsyncd.conf file. The + archetypal use is to offer public, anonymous, read-only + access to a software archive but as with + +

    + rsync can also run across a tunnel program such as + +

    + rsync may also be used locally, which is a degenerate case + with client and server connected across a unix-domain or + localhost socket. + + +

    + rsync runs on many varieties of Unix under both gcc and + native compilers, and also under Cygwin on Microsoft + platforms. There is somebody working on a VMS port. +

    +
    + + + + + + + + + + For example, you can imagine the server calculating and + caching the checksums, or the client calculating the delta + once and then sending it over UDP multicast to several + destinations, or uuencoded in email. Neither of these would + be straightforward to implement in the rsync codebase, but + all can be done with rdiff. + + + + + + + program, which + is a prototype of transparent integration of delta + compression into HTTP. +

    + +

    + rproxy implements delta compression of arbitrary HTTP + content, whether dynamic or static. For example, rproxy + gives very good compression on repeated visits to portal + sites such as Slashdot, since it can transmit only the + sections of the page modified since last time it was + viewed. Regular HTTP caches, by contrast, must either hit + on the whole page, or reload the whole page, and therefore + do poorly on template-based dynamic content. + +

    + rproxy adds compatible extensions to HTTP headers, and can + pass through HTTP caches. It requires upstream and + downstream support, which at the moment means two + installations of rproxy, but in the future could conceivably + be integrated into Mozilla, Squid, Apache and similar + programs. + +

    + rproxy is packaged in Debian and is moderately useful for + people with slow links. + +

    + rproxy is not HTML-specific, but HTML is by far the most + common case of dynamic HTTP content that can be + delta-compressed. However, HTML documents tend to be fairly + small, and as connectivity improves they're becoming of + decreasing interest as a compression problem. + +

    + (My personal interest in this project declined significantly + when I went from a 56k6 modem to ADSL at home. I realize + many people in the world still have slow connections.) + +

    + There are some Internet Drafts from Jeffrey Mogul and others + that add similar delta compression based on the server + storing all possible deltas. Last time I looked, there did + not seem much interest in wide adoption of these proposals. + +

    + Documenting a protocol extension to the standard expected by + an RFC can be a lot of work. A beginning on this work has + been made for rproxy, but more experiments with the + implementation are needed before proposal as a standard. + +

    + rproxy is not being actively developed at the moment. + Obviously I cannot answer why every programmer in the world + does not work on this. Personally I think that developing + rsync itself, and then librsync/rdiff, is likely to be more + useful; I suspect other people interested in working in this + area might have similar thoughts. + +

    + I don't think there are any problems in the code or project + that would actively prevent anybody from working on it. I'd + be happy to hand over maintenance to somebody else. Ben + Elliston has expressed interest in looking after it in the + last couple of months, and possibly it will be more active + in the future. + + + + + + Introduction to Debian + +

    + Debian is a free operating system developed by the + cooperation of people all over the world. The most + important relation of rsync to Debian is copying of software + packages from developers to distribution servers to end + users. +

    + +

    + Many Debian users get their software by internet download + from a public server, rather than pre-installed on a machine + or on CD-ROM. +

    + +

    + Debian software archives include both software source, + binary packages for various architectures, and index + metadata, primarily the + +

    + Because of Debian's community development process, new + packages are released very often. On a typical day in + Debian's + +

    + Debian is quite different from other software distributions + in shipping so many packages so frequently. The BSDs (as I + understand it) base most of their development out of a CVS + or CVSup tree, which inherently distributes coarse-grained + deltas. Proprietary systems make binary releases, but much + less frequently. Possibly the development branches of other + distributions, such as Redhat "Rawhide" and Mandrake + "Cooker" are similar. +

    +
    + + + + + + apt-proxy +

    + Because rsync is less efficient than HTTP for transferring + compressed files, + + The original apt-proxy was written by Paul 'Rusty' Russel. + The current maintainer is Chris Halls. +

    +
    +
    + + + + + Open Issues + + + + + + +

    + Although delta compression is not possible, rsync is still a + very useful tool for mirroring compressed files. The + efficient network protocol mean that it will generally be + slightly faster and use less traffic than HTTP or FTP. + +

    + There is a patch called + I believe that this will merge into the upstream + The scheme for containing changes is not specific to rsync, + and might also be useful with + This scheme relies on the output file being determined only + by the input file. As far as I know compression schemes + like gzip, lzw, and bzip2 are deterministic in this way. + (GnuPG, for example, is not, since it uses a random session + key.) + +

    + Incidentally this would allow more efficient deltas between + Debian ISO images. +

    + +

    + Alternatively, you could distribute an uncompressed package + tree that would rsync efficiently, but since the gzip patch + should merge soon this seems unnecessary. + +

    + There is also the interesting possibility of using + + It could be possible to make an equivalent patch to + + A patch to gzip which implements this behaviour is available + from rsync CVS. + + + + + + + If it is, then I think we should fix the problems, rather + than invent a new system from scratch. I think the + scalability problems are accidents of the current codebase, + rather than anything inherent in the design. +

    +
    + + + + + + Some projects (Apache 2.0, Mozilla, ...) choose to do this; + some concentrate on piecemeal improvement (Linux). + +

    + As Joel Spolsky recently observed, throwing the code out to + start from scratch always seems like a nice idea, but rarely + works out. Essentially you are opting for the devil you + don't know, but he definitely exists. + +

    + Having dealt with some fairly crufty old code I sometimes + feel like doing this in rsync, but I can also see the + arguments against it. Starting from a new codebase would + bring in plenty of new bugs (including security bugs), + orphan existing users, and halt progress until it caught up + to the same level. It would require carefully documenting + all the existing internal protocols and behaviours, which is + a nice idea but not clearly a good use of the developers' + time. + +

    + If you throw out the code, you have the questions of whether + or not to keep the current network protocol, and whether or + not to keep the current command-line option. + +

    + Not being able to talk to old remote versions would be a + real pain, and would certainly slow adoption. (Think of how + unpopular SSH2 was when it wanted to be incompatible with + SSH1, but use the same TCP port.) On the other hand, + keeping the same network protocol limits your ability to + redesign things. + + +

    + There is some cruft in the command line syntax, but throwing + it out would also break everybody's scripts, mental maps, + and documentation. + + +

    + Possibly rsync 3.0 will be largely rewritten. + + + + + + + Debian's goals for rsync have to be balanced against the + requirements of other users (including the mirror sites that + distribute Debian) and the limited development resources. + +

    + As of April 2002, the 2.5.5 release is current and 2.5.6 is + pending. These versions seem to work quite well and we + encourage people to upgrade from stable 2.4.6 and previous + versions. It is very important to the developers to + establish a stable base release before starting out on new + development. +

    + +

    + A number of enhancements are planned for the 2.6 series. + Some of them are discussed below, and more details can be + found in the + If you want rsync to progress faster, the best thing you can + do is find reproducible failure cases and report them well, + or to help us write a regression test suite. + +

    + Nobody is very actively working on rproxy or librsync as far + as I know. Personally at the moment I feel supporting rsync + itself is more useful. + + + + + + + + Goswin Brederlow has a nice proposal that allows rsync + compression using a regular HTTP server. It depends only + upon rsyncable archives, and development of new client and + support programs. + +

    + The idea, as I understand it, is that the checksums for each + package should be pre-calculated and stored on the HTTP + server along with the files they describe. (The checksum + files should be on the order of a thousand times smaller + than the archive file.) + +

    + The client program will download the checksum file over HTTP + if it exists. The client can then search for common blocks + in the local file, and therefore determine what new regions + it must fetch from the server. The client then sends an + HTTP request using the optional + This scheme has the great advantage that the server is + entirely passive, and only needs to support standard HTTP. + The checksum files can be generated once for each package + either during upload, or by the administrator of a + particular server. + +

    + (This is not the same as rproxy, which uses a different + encoding and requires upstream support, but can + transparently compress any request without requiring + signature files on the server.) + +

    + This scheme could be fairly easily built on top of rdiff or + librsync. + +

    + I think this sounds like a very promising scheme. I think + rsync might still be better for people who want to copy + large trees, but for users apt-getting single packages this + would be a simple way to get delta-compression in, pending + only --rsyncable files. + + +

    + I'm not sure if all HTTP servers currently handle + Because it requires a special client, and special checksum + files stored on the server it has a wider impact than just + using rsync. It ought to be done in a non-Debian-specific + way. Rather than adding knowledge about the deltas into + apt-get itself, we ought to make a separate tool which + downloads a delta over HTTP. + + + + + + + + rsync only compares files with the exact same + name + +

    + Even for uncompressed files, rsync will not currently try to + use + Paul Russell has contributed a patch which adds a heuristic + to detect this. It will probably be merged in 2.6. +

    +
    + + + + + rsync traverses the entire directory tree before it begins + copying files. On machines with little virtual memory and a + lot of files to copy this can be a problem. + +

    + Some problems in this area have been fixed in the 2.5 + series. Other solutions involving internal restructuring of + the + We are not likely to change the general approach of + traversing the tree up-front in the near future, because it + is tightly tied to the network protocol. It might be + attempted in the 3.0 timeframe, but it is not entirely clear + that it is really a problem. + + + + + + rsync hangs? + +

    + There were a number of deadlock bugs in the past. We + believe they are all fixed in 2.5.5, but would welcome good + bug reports to the contrary. Some of them were in fact + Linux and Solaris kernel bugs, but they've been fixed by the + relevant parties quite some time ago. +

    +
    + + + + + Reduced network usage is not justified by increased CPU + usage + + +

    + The balance point will vary for each administrator. It is + hard to answer this universally. The balance is very + different in countries other than the US. +

    + +

    + I suspect CPU cycles are falling in price faster than + network bandwidth, and for many people unused CPU cycles are + wasted but network packets cost money. +

    +
    + + + + + + rsync with --whole-file (to turn off the + delta algorithm and be more comparable to ftp) floods a + 100Mbps link. If you have something faster than that to + your upstream Debian mirror, you're a lucky person. +

    + +

    + It is not inherently impossible to make rsync use + sendfile() and mmap(); it's just that most + people don't need them. (Apache did not use them by default + either last time I looked, but that doesn't mean it's not + fast enough for most people.) +

    +
    + + + + + rsync is not actively developed + +

    + This was a pretty fair accusation a year ago, but I don't + think it is true anymore. We have made five releases this + year, and the + Mail is answered promptly. The web site has been revised + and cleaned. We're working on a regression test suite, code + cleanup and documentation, and other quality issues. + +

    + I'm the most active developer at the moment, but there are a + number of other people with experience in the code who + regularly commit changes or send patches. + +

    + The stability expectations of our users require a somewhat + disciplined approach to accepting patches, but I don't think + that's a bad thing. The goal is to iterate through + + The Debian rsync package maintainer, Phil Hands, has been + somewhat inactive, but apparently he's back now. Colin + Walters has been helping out. + +

    + If you have patches that were dropped, please send them + through again. + + + + + + Servers should cache deltas + +

    + This also smells like premature optimization. It might be + useful, but it certainly seems less important than some + other tasks. + + + + + + Possible patent on rsync? + +

    + It has been suggested that some uses of rsync may conflict + with a US patent. I am not sure of the current situation so + I won't comment. I don't know of any suggestion that the + rsync program as it currently exists infringes any patent. +

    + +

    + I do know that rsync has been in use for many years by large + and small organization with no complaints of patent + infringement. +

    + +

    + I would point out that linked lists, mark-and-copy garbage + collection, and the Tab key are all patented too. Somebody + who always carefully checked first for software patents + would never write anything at all. +

    + +
    + + + + Debian should store more metadata outside of + the package file + +

    + I can see some reasons why this might be a good idea, but + it doesn't seem particularly relevant to rsync either way. +

    +
    + + + + + Debian should distribute diffs or xdeltas + between different versions of their packages + +

    + A problem with this is that Debian releases packages very + often, and so the number of delta files is large. This will + be a problem for mirror sites who cannot carry all of the + delta files for history, although of course the client could + fall back to directly fetching the new package if the + relevant deltas are not present. +

    + +

    + Fetching a series of deltas which update the same area is + less efficient than calculating the deltas directly. +

    + +

    + Although this idea does not seem very practical for people + using + + + rsync should be used by + +

    + Even if there are few gains from compressing + + + + + + Revision history + + +

    +$Log: rsync-and-debian.sgml,v $ +Revision 1.10 2002/05/14 03:10:09 mbp +Add note about just transferring Package files. + +Revision 1.9 2002/05/14 02:35:43 mbp +Update notes on apt-proxy based on mail from Chris Halls. + +Revision 1.8 2002/04/12 02:04:15 mbp +More information about Goswin's idea, as I understand it. + +Revision 1.7 2002/04/12 01:39:52 mbp +Lots more details about rproxy. Thanks to Brian May for prompting me +to do this. + +Revision 1.6 2002/04/12 00:43:06 mbp +Fix revision history SGML stuff. + +Revision 1.5 2002/04/12 00:24:50 mbp +Fix mailing list name. +Add revision history. + +

    +
    +
    +
    \ No newline at end of file diff --git a/rsync-web/rsync-debug b/rsync-web/rsync-debug new file mode 100644 index 00000000..adb6ee5f --- /dev/null +++ b/rsync-web/rsync-debug @@ -0,0 +1,10 @@ +#!/bin/sh + +ulimit -c unlimited + +# Some systems have "truss" or "tusc" instead of "strace". +# The -f option tells strace to follow children too. +# The -t option asks for timestamps. +# The -s 1024 option increases the string decoding limit per function call. +# The -o option tells strace where to send its output. +strace -f -t -s 1024 -o /tmp/rsync-$$.out rsync "${@}" diff --git a/rsync-web/rsync_wrapper.pl b/rsync-web/rsync_wrapper.pl new file mode 100644 index 00000000..b760a977 --- /dev/null +++ b/rsync-web/rsync_wrapper.pl @@ -0,0 +1,471 @@ +# __ +# /\ \ From the mind of +# / \ \ +# / /\ \ \_____ Lee Eakin +# / \ \ \______\ or +# / /\ \ \/____ / +# \ \ \ \____\/ / Wrapper module for the rsync program +# \ \ \/____ / rsync can be found at http://rsync.samba.org/rsync/ +# \ \____\/ / +# \/______/ + +package Rsync; +require 5.004; + +use FileHandle; +use IPC::Open3 qw(open3); +use Carp; + +use strict; + +=head1 NAME + +Rsync - perl module interface to B http://rsync.samba.org/rsync/ + +=head1 SYNOPSIS + +use Rsync; + +$obj = Rsync->new(qw(-az C<-e> /usr/local/bin/ssh + --rsync-path /usr/local/bin/rsync)); + +$obj->exec(qw(localdir rhost:remdir)) + or warn "rsync failed\n"; + +=head1 DESCRIPTION + +Perl Convenience wrapper for B program. Written for B 2.3.1 but +should perform properly with most versions. + +=cut + +# options from the rsync man pae +###### Boolean flags ###### +# -h, --help show this help screen +# -v, --verbose increase verbosity +# -q, --quiet decrease verbosity +# -c, --checksum always checksum +# -a, --archive archive mode +# -r, --recursive recurse into directories +# -R, --relative use relative path names +# -b, --backup make backups (default ~ suffix) +# -u, --update update only (don't overwrite newer files) +# -l, --links preserve soft links +# -L, --copy-links treat soft links like regular files +# --copy-unsafe-links copy links outside the source tree +# --safe-links ignore links outside the destination tree +# -H, --hard-links preserve hard links +# -p, --perms preserve permissions +# -o, --owner preserve owner (root only) +# -g, --group preserve group +# -D, --devices preserve devices (root only) +# -t, --times preserve times +# -S, --sparse handle sparse files efficiently +# -n, --dry-run show what would have been transferred +# -W, --whole-file copy whole files, no incremental checks +# -x, --one-file-system don't cross filesystem boundaries +# -C, --cvs-exclude auto ignore files in the same way CVS does +# RCS SCCS CVS CVS.adm RCSLOG cvslog.* tags TAGS +# .make.state .nse_depinfo *~ #* .#* ,* *.old *.bak +# *.BAK *.orig *.rej .del-* *.a *.o *.obj *.so *.Z +# *.elc *.ln core +# --delete delete files that don't exist on the sending side +# --delete-excluded also delete excluded files on the receiving side +# --partial keep partially transferred files +# --force force deletion of directories even if not empty +# --numeric-ids don't map uid/gid values by user/group name +# -I, --ignore-times don't exclude files that match length and time +# --size-only only use file size when determining if a file +# should be transferred +# -z, --compress compress file data +# --version print version number +# --daemon run as a rsync daemon +# --stats give some file transfer stats +# --progress show progress during transfer +###### scalar values ###### +# --csum-length=LENGTH <=16 bit md4 checksum size +# -B, --block-size=SIZE checksum blocking size (default 700) +# --timeout=TIME set IO timeout in seconds +# --port=PORT specify alternate rsyncd port number +# -e, --rsh=COMMAND specify rsh replacement +# -T, --temp-dir=DIR create temporary files in directory DIR +# --compare-dest=DIR also compare destination files relative to DIR +# --exclude-from=FILE exclude patterns listed in FILE +# --include-from=FILE don't exclude patterns listed in FILE +# --config=FILE specify alternate rsyncd.conf file +# --password-file=FILE get password from FILE +# --log-format=FORMAT log file transfers using specified format +# --suffix=SUFFIX override backup suffix +# --rsync-path=PATH specify path to rsync on the remote machine +###### array values ###### +# --exclude=PATTERN exclude files matching PATTERN +# --include=PATTERN don't exclude files matching PATTERN +# + +=over 4 + +=item Rsync::new + +$obj = new Rsync; + + or + +$obj = Rsync->new; + + or + +$obj = Rsync->new(@options); + +=back + +Create an Rsync object. Any options passed at creation are stored in +the object as defaults for all future exec call on that object. Options +are the same as those in L with the addition of +--path-to-rsync which can be used to override the hardcoded default of +/usr/local/bin/rsync, and --debug which causes the module functions to +print some debugging information to STDERR. + +=cut + +sub new { + my $class=shift; + + # seed the options hash, booleans, scalars, excludes, data, + # status, stderr/stdout storage for last exec + my $self={ + # the full path name to the rsync binary + 'path-to-rsync' => '/usr/local/bin/rsync', + # these are the boolean flags to rsync, all default off, including them + # in the args list turns them on + 'flag' => {qw( + --archive 0 --backup 0 --checksum 0 + --compress 0 --copy-links 0 --copy-unsafe-links 0 + --cvs-exclude 0 --daemon 0 --delete 0 + --delete-excluded 0 --devices 0 --dry-run 0 + --force 0 --group 0 --hard-links 0 + --help 0 --ignore-times 0 --links 0 + --numeric-ids 0 --one-file-system 0 --owner 0 + --partial 0 --perms 0 --progress 0 + --quiet 0 --recursive 0 --relative 0 + --safe-links 0 --size-only 0 --sparse 0 + --stats 0 --times 0 --update 0 + --verbose 0 --version 0 --whole-file 0 + )}, + # these have simple scalar args we cannot easily check + 'scalar' => {qw( + --block-size 0 --compare-dest 0 --config 0 + --csum-length 0 --exclude-from 0 --include-from 0 + --log-format 0 --password-file 0 --port 0 + --rsh 0 --rsync-path 0 --suffix 0 + --temp-dir 0 --timeout 0 + )}, + # these can be specified multiple times and are additive, the doc also + # specifies that it is an ordered list so we must preserve that order + 'exclude' => [], + # source/destination path names and hostnames + 'data' => [], + # return status from last exec + 'status' => 0, + 'realstatus' => 0, + # whether or not to print debug statements + 'debug' => 0, + # stderr from last exec in array format (messages from remote rsync proc) + 'err' => [], + # stdout from last exec in array format (messages from local rsync proc) + 'out' => [], + }; + if (@_) { + &defopts($self,@_) or return undef; + } + bless $self, $class; +} + +=over 4 + +=item Rsync::defopts + +defopts $obj @options; + + or + +$obj->defopts(@options); + +=back + +Set default options for future exec calls for the object. See L +for a complete list of valid options. This is really the internal +function that B calls but you can use it too. Presently there is no way +to turn off the boolean options short of creating another object, but if it is +needed and the B guys don't use it, I may add hooks to let + and ++ or a +leading no- toggle it back off similar to B (the GNU way). + +=cut + +sub defopts { + my $self=shift; + my @opts=@_; + + # need a conversion table in case someone uses the short options + my %short=(qw( + -B --block-size -C --cvs-exclude -D --devices -H --hard-links + -I --ignore-times -L --copy-links -R --relative -T --temp-dir + -W --whole-file -a --archive -b --backup -c --checksum + -e --rsh -g --group -h --help -l --links + -n --dry-run -o --owner -p --perms -q --quiet + -r --recursive -s --sparse -t --times -u --update + -v --verbose -x --one-file-system -z --compress + )); + while (my $opt=shift @opts) { + my $arg; + print(STDERR "setting debug flag\n"),$self->{debug}=1,next + if $opt eq '--debug'; + print STDERR "processing option: $opt\n" if $self->{debug}; + if ($opt=~/^-/) { + # handle short opts first + if ($opt=~/^-(\w+)$/) { + foreach (split '',$1) { + print STDERR "short option: -$_\n" if $self->{debug}; + if (exists $short{'-'.$_}) { + $opt=$short{'-'.$_}; + # convert it to the long form + $opt=$short{$opt} if exists $short{$opt}; + # handle the 3 short opts that require args + $self->{scalar}{$opt}=shift(@opts),next if (/^[BeT]$/); + # handle the rest + $self->{flag}{$opt}=1,next if exists $self->{flag}{$opt}; + } + carp "$opt - unknown option\n"; + return undef; + } + } + # handle long opts with = args + if ($opt=~/^(--\w+[\w-]*)=(.*)$/) { + print STDERR "splitting longopt: $opt ($1 $2)\n" if $self->{debug}; + ($opt,$arg)=($1,$2); + } + # handle boolean flags + $self->{flag}{$opt}=1,next if exists $self->{flag}{$opt}; + # handle simple scalars + $self->{scalar}{$opt}=($arg || shift @opts),next + if exists $self->{scalar}{$opt}; + # handle excludes + if ($opt eq '--exclude') { + $arg||=shift @opts; + # if they sent a reset, we will too + $self->{exclude}=[],next if $arg eq '!'; + # otherwise add it to the list + push @{$self->{exclude}},$arg; + next; + } + # to preserve order we store both includes and excludes in the same + # array. We use the leading '+ ' (plus space) trick from the man + # page to accomplish this. + if ($opt eq '--include') { + $arg||=shift @opts; + # first check to see if this is really an exclude + push(@{$self->{exclude}},$arg),next if $arg=~s/^- //; + # next see if they sent a reset, if they did, we will too + $self->{exclude}=[],next if $arg eq '!'; + # if it really is an include, fix it first, since we use exclude + $arg='+ '.$arg unless $arg=~/^\+ /; + push @{$self->{exclude}},$arg; + next; + } + # handle our special case to override hard-coded path to rsync + $self->{'path-to-rsync'}=($arg || shift @opts),next + if $opt eq '--path-to-rsync'; + # if we get this far nothing matched so it must be an error + carp "$opt - unknown option\n"; + return undef; + } else { # must be data (source/destination info) + print STDERR "adding to data array: $opt\n" if $self->{debug}; + push(@{$self->{data}},$opt); + } + } + 1; +} + +=over 4 + +=item Rsunc::exec + +exec $obj @options or warn "rsync failed\n"; + + or + +$obj->exec(@options) or warn "rsync failed\n"; + +=back + +This is the function that does the real work. Any options passed to this +routine are appended to any pre-set options and are not saved. They effect +the current execution of B only. It returns 1 if the return status was +zero (or true), if the B return status was non-zero it returns undef and +stores the return status. You can examine the return status from rsync and +any output to stdout and stderr with the functions listed below. + +=cut + +sub exec { + my $self=shift; + my @cmd=($self->{'path-to-rsync'}); + + foreach (sort keys %{$self->{flag}}) { + push @cmd,$_ if $self->{flag}{$_}; + } + foreach (sort keys %{$self->{scalar}}) { + push @cmd,$_.'='.$self->{scalar}{$_} if $self->{scalar}{$_}; + } + foreach (@{$self->{exclude}}) { + push @cmd,'--exclude='.$_; + } + foreach (@{$self->{data}}) { + push @cmd,$_; + } + push @cmd,@_ if @_; + print STDERR "exec: @cmd\n" if $self->{debug}; + my $in=FileHandle->new; my $out=FileHandle->new; my $err=FileHandle->new; + my $pid=eval{ open3 $in,$out,$err,@cmd }; + if ($@) { + $self->{realstatus}=0; + $self->{status}=255; + $self->{err}=[$@,"Execution of rsync failed.\n"]; + return undef; + } + $in->close; # we don't use it and neither should rsync (at least not yet) + $self->{err}=[ $err->getlines ]; + $self->{out}=[ $out->getlines ]; + $err->close; + $out->close; + waitpid $pid,0; + $self->{realstatus}=$?; + $self->{status}=$?>>8; + return undef if $self->{status}; + return 1; +} + +=over 4 + +=item Rsync::status + +$rval = status $obj; + + or + +$rval = $obj->status; + +=back + +Returns the status from last B call right shifted 8 bits. + +=cut + +sub status { + my $self=shift; + return $self->{status}; +} + +=over 4 + +=item Rsync::realstatus + +$rval = realstatus $obj; + + or + +$rval = $obj->realstatus; + +=back + +Returns the real status from last B call (not right shifted). + +=cut + +sub realstatus { + my $self=shift; + return $self->{realstatus}; +} + +=over 4 + +=item Rsync::err + +$aref = err $obj; + + or + +$aref = $obj->err; + +=back + +Returns an array or a reference to the array containing all output to stderr +from the last B call. B sends all messages from the remote +B process to stderr. This functions purpose is to make it easier for +you to parse that output for appropriate information. + +=cut + +sub err { + my $self=shift; + return(wantarray ? @{$self->{err}} : $self->{err}); +} + +=over 4 + +=item Rsync::out + +$aref = out $obj; + + or + +$aref = $obj->out; + +=back + +Similar to the B function, this returns an array or a reference to the +array containing all output to stdout from the last B call. B +sends all messages from the local B process to stdout. + +=cut + +sub out { + my $self=shift; + return(wantarray ? @{$self->{out}} : $self->{out}); +} + +=head1 Author + +Lee Eakin Eleakin@nostrum.comE + +=head1 Credits + +Gerard Hickey C + +Russ Allbery C + +Graham Barr C + +Andrew Tridgell and Paul Mackerras C + +John Steele Esteele@nostrum.comE + +Philip Kizer Epckizer@nostrum.comE + +Larry Wall C + +I borrowed many clues on wrapping an external program from the PGP modules, +and I would not have had such a useful tool to wrap except for the great work +of the B authors. Thanks also to Graham Barr, the author of the libnet +modules and many others, for looking over this code. Of course I must mention +the other half of my brain, John Steele, and his good friend Philip Kizer for +finding B and bringing it to my attention. And I would not have been +able to enjoy writing useful tools if not for the creator of the B +language. + +=head1 Copyrights + + Copyleft (l) 1999, by Lee Eakin + +=cut + +1; diff --git a/rsync-web/security.html b/rsync-web/security.html new file mode 100644 index 00000000..fedcffcb --- /dev/null +++ b/rsync-web/security.html @@ -0,0 +1,284 @@ + + + +rsync + + + + +

    Rsync Security Advisories

    + +

    You should install a security fix for rsync when the rync you are running is:

      +
    1. older than 3.2.5 and pulling from an untrusted server +
    2. older than 3.2.5 and using the bundled zlib +
    3. older than 3.1.3 with --xattrs enabled +
    4. older than 3.1.3 with a writable rsync daemon +
    5. older than 2.6.6 +
    + +


    +

    Improved file-list validation in 3.2.5

    +August 14th, 2022 + +

    If you are running an rsync older than 3.2.5 and pulling files from an +untrusted server, upgrade to 3.2.5 to get some added file-list validation rules +that should prevent the sender from sneaking in extra top-level arguments +and/or including files/dirs that should have been filtered out by the client's +filter rules. Fixes CVE-2022-29154. + +


    +

    Zlib memory corruption bug in rsync 2.6.6 - 3.2.4

    +August 14th, 2022 + +

    If your rsync is configured to use the bundled zlib, you should upgrade to +3.2.5 which contains the official zlib fix for a buffer overrun bug that was +detailed in CVE-2022-37434. While you're at it, be sure to update your system's +zlib. + +


    +

    Zlib memory corruption bug in rsync 2.6.6 - 3.2.3

    +April 15th, 2022 + +

    If your rsync is configured to use the bundled zlib, you should upgrade to +3.2.4 which contains the official zlib fix for a memory corruption bug that was +detailed in CVE-2018-25032. While you're at it, be sure to update your system's +zlib. + +


    +

    File-list validation in 3.1.3

    +January 28st, 2018 + +

    If you are using a version of rsync older than 3.1.3 as a client and +receiving xattrs from an rsync server that you might not fully trust, a +malicious (modified) server could send a non-null-terminated xattr name to +overflow the xattr read buffer. + +

    If you are running a writable rsync daemon older than 3.1.3, you should add +a rule "refuse options = protect-args" if you don't fully trust the users who +are sending you files. + +


    +

    File-list validation in 3.1.2

    +December 21st, 2015 + +

    If you're using a version of rsync older than 3.1.2 as a client and +receiving files from an rsync server that you might not fully trust, this +version adds extra checking to the file list to prevent the sender from +tweaking the paths and/or the transfer requests in a way that could cause +a file to be received outside the transfer destination. + +


    +

    Xattr security fix in 3.0.2

    +April 8th, 2008 + +

    If you're using a version of rsync from 2.6.9 to 3.0.1 that has extended +attribute (xattr) support enabled, you should upgrade to 3.0.2 to avoid a +potential buffer overflow problem. All 3.x versions have the potential to +support xattrs (depending on OS availability and the configure options used), +but version 2.6.9 had to be patched for this support. You can run the command +"rsync --version" and look for the string "xattrs" (as long as it is not +"no xattrs") to see if your rsync is affected. + +

    For those running affected versions, there is +a +patch with the fix available. + +

    Those running a writable rsync daemon can opt to refuse the "xattrs" option +as a way to avoid the problem without an upgrade: + +

    refuse options = xattrs
    + +

    (If you already refuse options, be sure to append "xattrs" to your existing +config parameter rather than adding another refuse directive.) + +


    +

    Daemon security fixes in 3.0.0 (with patches for 2.6.9)

    +First published on November 28th, 2007;
    +Updated on December 16th, 2007;
    +Item 3 added on February 15th, 2008
    + +

    Three security advisories affect people who run a writable rsync +daemon: The first affects only those with "use chroot = no" (which is not a +very safe combination in general), the second affects a daemon that has +daemon-excluded files that are being hidden in a module's hierarchy, and +the third affects only those with "use chroot = yes". +Included are simple config-change suggestions that should help you to +avoid the security issues and patches that make things safer. +These advisories affect all rsync versions. + +

    1. Daemon advisory for "use chroot = no"

    + +

    If you are running a writable rsync daemon with "use chroot = no", there +is at least one way for someone to trick rsync into creating a symlink +that points outside of the module's hierarchy. + +

    This means that if you are allowing access from users who you don't +trust, that you should either figure out a way to turn on "use chroot", +or configure the daemon to refuse the "links" option (see "refuse +options" in the rsyncd.conf manpage) which will disable the ability of +the rsync module to receive symlinks. After doing so, you should also +check that any existing symlinks in the daemon hierarchy are safe. + +

    Starting with the 3.0.0-pre6 release, there is a new daemon parameter +available: "munge symlinks". This allows an rsync daemon to accept +symlinks and return them intact (with even a leading slash still there, +which is new for a non-chroot daemon), but will not allow the symlinks +to be used while they are in the daemon's hierarchy. + +

    For those running +2.6.9, there is +a +patch for 2.6.9 to implement this parameter. + +

    Any admin applying that patch should read the "munge symlinks" section +of the modified rsyncd.conf manpage for more information. You can also +read about this parameter in the +rsyncd.conf +manpage from a 3.x version. + +

    2. Daemon advisory for daemon excludes

    + +

    If you are running a writable rsync daemon that is using one of the +"exclude", "exclude from", or "filter" parameters in the rsyncd.conf file +to hide data from your users, you should be aware that there are tricks +that a user can play with symlinks and/or certain options that can allow +a user that knows the name of a hidden file to access it or overwrite it +(if file permissions allow that). + +

    You can avoid the symlink problem using the suggestions in the advisory +above. + +

    When a daemon has "use chroot = no" set , there was some buggy +exclude-checking for these options: --compare-dest, +--link-dest, --copy-dest, --partial-dir, +--backup-dir, --temp-dir, and +--files-from. These are fixed in the 3.0.0pre7 release. For +those running 2.6.9, there is a patch for +2.6.9 to fix these checks. + +

    Some of the above options can still cause problems if an excluded +sub-directory or filename is inside the option's directory hierarchy and the +names of a transferred file clashes with it. The affected options are the +various --*-dest options (of which only --link-dest +is particularly worrisome), +--backup-dir, and --partial-dir. + +

    You can avoid these sub-path problems by putting the following "refuse +options" setting into your rsyncd.conf file: + +

    refuse options = link-dest backup-dir partial-dir
    + +

    Those who aren't using an rsync with the latest exclude fixes may want to add +some of the other affected options as well. + +

    3. Daemon advisory for "use chroot = yes"

    + +

    If you are running a writable rsync daemon with "use chroot = yes", you +should take care that users cannot upload their own library files and attempt +to load them. + +

    Beginning with rsync 3.0.0pre10, you can specify an inside-chroot path that +makes the top of the transfer a subdirectory inside the chroot area, and that +automatically makes library loading occur outside the transfer area (assuming +you didn't pick an unwise subdirectory name for the transfer area and you +don't have symlinks that point outside the transfer area). + +

    If that solution is not good for you, the easiest way to protect your daemon +is to create some appropriate directories in the top of your module's path +hierarchy, such as "/etc", "/lib", and "/usr" (and any other top-level dirs +that might be in the load path), chown those directories to some other user +than the one that the module runs as (so that rsync will not be able to write +files there, assuming that it is not run as root), and then hide the dirs using +an exclude directive (either add a new one or extend your existing one): + +

    exclude = /etc /lib /usr
    + +

    Doing all that will assure you that no user will be able to use rsync to +upload a library that can be potentially loaded while rsync is attempting to +perform an action, such as translating a username. You can feel free to put +trusted libraries that you want rsync to access in the protected hierarchies, +as desired. + +

    Also available in rsync 3.0.0pre10 is a new daemon parameter that allows you +to avoid the accessing of user/group-name translation libraries by a chrooted +rsync: the "numeric ids" daemon parameter lets you turn on a forced +numeric-only mode. The default for a chroot module is to enable this +parameter, while the default for a non-chroot module is to disable it. + +

    For those running 2.6.9, there is a patch for +2.6.9 to add the "numeric ids" daemon config parameter. (The patch will +only apply cleanly if you've already applied the munge-symlinks diff mentioned +above.) + +


    +

    Xattr security fix in 2.6.8

    +April 22th, 2006 + +

    If you're using a version of rsync prior to 2.6.8 that was patched to +include extended attribute (xattr) support, you should upgrade to 2.6.8 or +later to avoid a potential buffer overflow problem. + +


    +

    Zlib security fix in 2.6.6

    +July 28th, 2005 + +

    If you're using a version of rsync prior to 2.6.6, there is a potential +null-pointer security bug in the zlib code. You can avoid its affect in an +rsync daemon situation by configuring rsync to refuse the "compress" option. + +


    +

    Daemon path-sanitizing fix in 2.6.3

    +August 12th, 2004 + +

    There is a path-sanitizing bug that affects daemon-mode in +rsync versions prior to 2.6.3, but only if "use chroot" is disabled. It +does not affect the normal send/receive filenames that specify what +files should be transferred (this is because these names happen to get +sanitized twice, and thus the second call removes any lingering leading +slash(es) that the first call left behind). It does affect certain +option paths that cause auxiliary files to be read or written. + +

    One potential fix that doesn't require recompiling rsync is to set +"use chroot = true" for all the modules in the rsyncd.conf file. + +


    +

    Daemon security fix in 2.6.1

    +April 26th, 2004 + +

    There is a security problem in all versions prior to 2.6.1 that affects only +people running a read/write daemon with "use chroot" disabled. If the user privs +of a module in the daemon config is anything above "nobody", you are at risk +of someone crafting an attack that could write a file outside of the module's +"path" setting (where all its files should be stored). Please either enable +chroot or upgrade to 2.6.1. People not running a daemon, running a read-only +daemon, or running a chrooted daemon are totally unaffected. + +


    +

    Memory overflow fix in 2.5.7

    +December 4th, 2003 + +

    Rsync versions prior to 2.5.7 contain a heap overflow vulnerability that +could be used to remotely run arbitrary code, but this only affects the use of +rsync as an "rsync daemon" (where rsync handles incoming socket connections, +typically on port 873). + + +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      diff --git a/rsync-web/susan.txt b/rsync-web/susan.txt new file mode 100644 index 00000000..2d8a67e3 --- /dev/null +++ b/rsync-web/susan.txt @@ -0,0 +1,31 @@ +I use rsync to backup my wifes home directory across a modem link each +night. The cron job looks like this + + #!/bin/sh + cd ~susan + { + echo + date + dest=~/backup/`date +%A` + mkdir $dest.new + find . -xdev -type f \( -mtime 0 -or -mtime 1 \) -exec cp -aPv "{}" + $dest.new \; + cnt=`find $dest.new -type f | wc -l` + if [ $cnt -gt 0 ]; then + rm -rf $dest + mv $dest.new $dest + fi + rm -rf $dest.new + rsync -Cavze ssh . samba:backup + } >> ~/backup/backup.log 2>&1 + + +note that most of this script isn't anything to do with rsync, it just +creates a daily backup of Susans work in a ~susan/backup/ directory so +she can retrieve any version from the last week. The last line does +the rsync of her directory across the modem link to the host +samba. Note that I am using the -C option which allows me to add +entries to .cvsignore for stuff that doesn't need to be backed up. + + + diff --git a/rsync-web/tech_report/footnode.html b/rsync-web/tech_report/footnode.html new file mode 100644 index 00000000..0c66b881 --- /dev/null +++ b/rsync-web/tech_report/footnode.html @@ -0,0 +1,134 @@ + + + + +Footnotes + + + + + + + + + + + +

    +
    ... bytes1 +
    We have found that + values of S between 500 and 1000 are quite good for most purposes +
    .
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +
    +
    ... size.2 +
    All the tests in this section were + carried out using rsync version 0.5 +
    .
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +
    +
    ... files.3 +
    The wall + clock time was approximately 2 minutes per run on a 50 MHz SPARC 10 + running SunOS, using rsh over loopback for communication. GNU diff + took about 4 minutes. +
    .
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +.
    +
    +
    +Andrew Tridgell +
    1998-11-09 +
    + + diff --git a/rsync-web/tech_report/images.aux b/rsync-web/tech_report/images.aux new file mode 100644 index 00000000..f23e5468 --- /dev/null +++ b/rsync-web/tech_report/images.aux @@ -0,0 +1 @@ +\relax diff --git a/rsync-web/tech_report/images.log b/rsync-web/tech_report/images.log new file mode 100644 index 00000000..4344b35f --- /dev/null +++ b/rsync-web/tech_report/images.log @@ -0,0 +1,106 @@ +This is TeX, Version 3.14159 (C version 6.1) (format=latex 97.10.16) 9 NOV 1998 20:10 +**./images.tex +(images.tex +LaTeX2e <1996/12/01> patch level 1 +Babel and hyphenation patterns for american, german, loaded. + +(/usr/lib/texmf/texmf/tex/latex/base/article.cls +Document Class: article 1996/10/31 v1.3u Standard LaTeX document class +(/usr/lib/texmf/texmf/tex/latex/base/size10.clo +File: size10.clo 1996/10/31 v1.3u Standard LaTeX file (size option) +) +\c@part=\count79 +\c@section=\count80 +\c@subsection=\count81 +\c@subsubsection=\count82 +\c@paragraph=\count83 +\c@subparagraph=\count84 +\c@figure=\count85 +\c@table=\count86 +\abovecaptionskip=\skip41 +\belowcaptionskip=\skip42 +\bibindent=\dimen102 +) (/usr/lib/texmf/texmf/tex/latex/graphics/color.sty +Package: color 1997/01/07 v1.0d Standard LaTeX Color (DPC) +(/usr/lib/texmf/texmf/tex/latex/config/color.cfg) +Package color Info: Driver file: dvips.def on input line 128. +(/usr/lib/texmf/texmf/tex/latex/graphics/dvips.def +File: dvips.def 1996/12/12 v3.0d Driver-dependant file (DPC,SPQR) +) (/usr/lib/texmf/texmf/tex/latex/graphics/dvipsnam.def +File: dvipsnam.def 1996/12/12 v3.0d Driver-dependant file (DPC,SPQR) +)) +\sizebox=\box26 +\lthtmlwrite=\write3 +No file images.aux. +LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 74. +LaTeX Font Info: ... okay on input line 74. +LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 74. +LaTeX Font Info: ... okay on input line 74. +LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 74. +LaTeX Font Info: ... okay on input line 74. +LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 74. +LaTeX Font Info: ... okay on input line 74. +LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 74. +LaTeX Font Info: ... okay on input line 74. +LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 74. +LaTeX Font Info: ... okay on input line 74. +latex2htmlLength hsize=349.0pt +latex2htmlLength vsize=682.0pt +latex2htmlLength hoffset=0.0pt +latex2htmlLength voffset=0.0pt +latex2htmlLength topmargin=0.0pt +latex2htmlLength topskip=0.00003pt +latex2htmlLength headheight=0.0pt +latex2htmlLength headsep=0.0pt +latex2htmlLength parskip=0.0pt plus 1.0pt +latex2htmlLength oddsidemargin=53.0pt +latex2htmlLength evensidemargin=53.0pt +LaTeX Font Info: External font `cmex10' loaded for size +(Font) <7> on input line 94. +LaTeX Font Info: External font `cmex10' loaded for size +(Font) <5> on input line 94. +l2hSize :tex2html_wrap_inline88:6.45831pt::0.0pt::6.43404pt. +[1 + + +] +l2hSize :tex2html_wrap_inline90:7.44444pt::7.44444pt::6.18402pt. +[2 + + +] +l2hSize :displaymath158:32.38896pt::0.0pt::349.0pt. +[3 + + +] +l2hSize :displaymath160:32.38896pt::0.0pt::349.0pt. +[4 + + +] +l2hSize :tex2html_wrap_inline166:7.33331pt::7.33331pt::39.55894pt. +[5 + + +] +l2hSize :displaymath170:14.5pt::0.0pt::349.0pt. +[6 + + +] +l2hSize :displaymath172:14.5pt::0.0pt::349.0pt. +[7 + + +] (images.aux) ) +Here is how much of TeX's memory you used: + 518 strings out of 10906 + 6196 string characters out of 72186 + 46914 words of memory out of 262141 + 3454 multiletter control sequences out of 9500 + 3640 words of font info for 14 fonts, out of 150000 for 255 + 14 hyphenation exceptions out of 607 + 23i,5n,19p,182b,215s stack positions out of 300i,40n,60p,3000b,4000s + +Output written on images.dvi (7 pages, 2296 bytes). diff --git a/rsync-web/tech_report/images.pl b/rsync-web/tech_report/images.pl new file mode 100644 index 00000000..6fb022fb --- /dev/null +++ b/rsync-web/tech_report/images.pl @@ -0,0 +1,48 @@ +# LaTeX2HTML 98.1p1 release (March 2nd, 1998) +# Associate images original text with physical files. + + +$key = q/{inline}alpha{inline}MSF=1.6;AAT;/; +$cached_env_img{$key} = q|$\alpha$|; + +$key = q/{displaymath}a(k+1,l+1)=(a(k,l)-X_k+X_l+1)bmodM{displaymath}MSF=1.6;AAT;/; +$cached_env_img{$key} = q|\begin{displaymath}a(k+1,l+1) = (a(k,l) - X_k + X_{l+1}) \bmod M \end{displaymath}|; + +$key = q/{displaymath}a(k,l)=(sum_i=k^lX_i)bmodM{displaymath}MSF=1.6;AAT;/; +$cached_env_img{$key} = q|\begin{displaymath}a(k,l) = (\sum_{i=k}^l X_i) \bmod M \end{displaymath}|; + +$key = q/{displaymath}b(k,l)=(sum_i=k^l(l-i+1)X_i)bmodM{displaymath}MSF=1.6;AAT;/; +$cached_env_img{$key} = q|\begin{displaymath}b(k,l) = (\sum_{i=k}^l (l-i+1)X_i) \bmod M \end{displaymath}|; + +$key = q/{inline}X_kldotsX_l{inline}MSF=1.6;AAT;/; +$cached_env_img{$key} = q|$X_k \ldots X_l$|; + +$key = q/{displaymath}b(k+1,l+1)=(b(k,l)-(l-k+1)X_k+a(k+1,l+1))bmodM{displaymath}MSF=1.6;AAT;/; +$cached_env_img{$key} = q|\begin{displaymath}b(k+1,l+1) = (b(k,l) - (l-k+1) X_k + a(k+1,l+1)) \bmod M \end{displaymath}|; + +$key = q/{inline}beta{inline}MSF=1.6;AAT;/; +$cached_env_img{$key} = q|$\beta$|; + +1; + diff --git a/rsync-web/tech_report/images.tex b/rsync-web/tech_report/images.tex new file mode 100644 index 00000000..61a3d9e5 --- /dev/null +++ b/rsync-web/tech_report/images.tex @@ -0,0 +1,140 @@ +\batchmode +\documentclass[a4paper]{article} +\makeatletter +\usepackage[dvips]{color} +\pagecolor[gray]{.7} + + +\makeatletter +\count@=\the\catcode`\_ \catcode`\_=8 +\newenvironment{tex2html_wrap}{}{} \catcode`\_=\count@ +\makeatother +\let\mathon=$ +\let\mathoff=$ +\ifx\AtBeginDocument\undefined \newcommand{\AtBeginDocument}[1]{}\fi +\newbox\sizebox +\setlength{\hoffset}{0pt}\setlength{\voffset}{0pt} +\addtolength{\textheight}{\footskip}\setlength{\footskip}{0pt} +\addtolength{\textheight}{\topmargin}\setlength{\topmargin}{0pt} +\addtolength{\textheight}{\headheight}\setlength{\headheight}{0pt} +\addtolength{\textheight}{\headsep}\setlength{\headsep}{0pt} +\setlength{\textwidth}{349pt} +\newwrite\lthtmlwrite +\makeatletter +\let\realnormalsize=\normalsize +\global\topskip=2sp +\def\preveqno{}\let\real@float=\@float \let\realend@float=\end@float +\def\@float{\let\@savefreelist\@freelist\real@float} +\def\end@float{\realend@float\global\let\@freelist\@savefreelist} +\let\real@dbflt=\@dbflt \let\end@dblfloat=\end@float +\let\@largefloatcheck=\relax +\def\@dbflt{\let\@savefreelist\@freelist\real@dbflt} +\def\adjustnormalsize{\def\normalsize{\mathsurround=0pt \realnormalsize + \parindent=0pt\abovedisplayskip=0pt\belowdisplayskip=0pt}\normalsize}% +\def\lthtmltypeout#1{{\let\protect\string\immediate\write\lthtmlwrite{#1}}}% +\newcommand\lthtmlhboxmathA{\adjustnormalsize\setbox\sizebox=\hbox\bgroup}% +\newcommand\lthtmlvboxmathA{\adjustnormalsize\setbox\sizebox=\vbox\bgroup% + \let\ifinner=\iffalse }% +\newcommand\lthtmlboxmathZ{\@next\next\@currlist{}{\def\next{\voidb@x}}% + \expandafter\box\next\egroup}% +\newcommand\lthtmlmathtype[1]{\def\lthtmlmathenv{#1}}% +\newcommand\lthtmllogmath{\lthtmltypeout{l2hSize % +:\lthtmlmathenv:\the\ht\sizebox::\the\dp\sizebox::\the\wd\sizebox.\preveqno}}% +\newcommand\lthtmlfigureA[1]{\let\@savefreelist\@freelist + \lthtmlmathtype{#1}\lthtmlvboxmathA}% +\newcommand\lthtmlfigureZ{\lthtmlboxmathZ\lthtmllogmath\copy\sizebox + \global\let\@freelist\@savefreelist}% +\newcommand\lthtmldisplayA[1]{\lthtmlmathtype{#1}\lthtmlvboxmathA}% +\newcommand\lthtmldisplayB[1]{\edef\preveqno{(\theequation)}% + \lthtmldisplayA{#1}\let\@eqnnum\relax}% +\newcommand\lthtmldisplayZ{\lthtmlboxmathZ\lthtmllogmath\lthtmlsetmath}% +\newcommand\lthtmlinlinemathA[1]{\lthtmlmathtype{#1}\lthtmlhboxmathA \vrule height1.5ex width0pt }% +\newcommand\lthtmlinlineA[1]{\lthtmlmathtype{#1}\lthtmlhboxmathA}% +\newcommand\lthtmlinlineZ{\egroup\expandafter\ifdim\dp\sizebox>0pt % + \expandafter\centerinlinemath\fi\lthtmllogmath\lthtmlsetinline} +\newcommand\lthtmlinlinemathZ{\egroup\expandafter\ifdim\dp\sizebox>0pt % + \expandafter\centerinlinemath\fi\lthtmllogmath\lthtmlsetmath} +\def\lthtmlsetinline{\hbox{\vrule width.1em\vtop{\vbox{% + \kern.1em\copy\sizebox}\ifdim\dp\sizebox>0pt\kern.1em\else\kern.3pt\fi + \ifdim\hsize>\wd\sizebox \hrule depth1pt\fi}}} +\def\lthtmlsetmath{\hbox{\vrule width.1em\vtop{\vbox{% + \kern.1em\kern0.8 pt\hbox{\hglue.17em\copy\sizebox\hglue0.8 pt}}\kern.3pt% + \ifdim\dp\sizebox>0pt\kern.1em\fi \kern0.8 pt% + \ifdim\hsize>\wd\sizebox \hrule depth1pt\fi}}} +\def\centerinlinemath{%\dimen1=\ht\sizebox + \dimen1=\ifdim\ht\sizebox<\dp\sizebox \dp\sizebox\else\ht\sizebox\fi + \advance\dimen1by.5pt \vrule width0pt height\dimen1 depth\dimen1 + \dp\sizebox=\dimen1\ht\sizebox=\dimen1\relax} + +\def\lthtmlcheckvsize{\ifdim\ht\sizebox<\vsize\expandafter\vfill + \else\expandafter\vss\fi}% +\makeatletter \tracingstats = 1 + + +\begin{document} +\pagestyle{empty}\thispagestyle{empty}% +\lthtmltypeout{latex2htmlLength hsize=\the\hsize}% +\lthtmltypeout{latex2htmlLength vsize=\the\vsize}% +\lthtmltypeout{latex2htmlLength hoffset=\the\hoffset}% +\lthtmltypeout{latex2htmlLength voffset=\the\voffset}% +\lthtmltypeout{latex2htmlLength topmargin=\the\topmargin}% +\lthtmltypeout{latex2htmlLength topskip=\the\topskip}% +\lthtmltypeout{latex2htmlLength headheight=\the\headheight}% +\lthtmltypeout{latex2htmlLength headsep=\the\headsep}% +\lthtmltypeout{latex2htmlLength parskip=\the\parskip}% +\lthtmltypeout{latex2htmlLength oddsidemargin=\the\oddsidemargin}% +\makeatletter +\if@twoside\lthtmltypeout{latex2htmlLength evensidemargin=\the\evensidemargin}% +\else\lthtmltypeout{latex2htmlLength evensidemargin=\the\oddsidemargin}\fi% +\makeatother +\stepcounter{section} +\stepcounter{section} +{\newpage\clearpage +\lthtmlinlinemathA{tex2html_wrap_inline88}% +$\alpha$% +\lthtmlinlinemathZ +\hfill\lthtmlcheckvsize\clearpage} + +{\newpage\clearpage +\lthtmlinlinemathA{tex2html_wrap_inline90}% +$\beta$% +\lthtmlinlinemathZ +\hfill\lthtmlcheckvsize\clearpage} + +\stepcounter{section} +{\newpage\clearpage +\lthtmldisplayA{displaymath158}% +\begin{displaymath}a(k,l) = (\sum_{i=k}^l X_i) \bmod M \end{displaymath}% +\lthtmldisplayZ +\hfill\lthtmlcheckvsize\clearpage} + +{\newpage\clearpage +\lthtmldisplayA{displaymath160}% +\begin{displaymath}b(k,l) = (\sum_{i=k}^l (l-i+1)X_i) \bmod M \end{displaymath}% +\lthtmldisplayZ +\hfill\lthtmlcheckvsize\clearpage} + +{\newpage\clearpage +\lthtmlinlinemathA{tex2html_wrap_inline166}% +$X_k \ldots X_l$% +\lthtmlinlinemathZ +\hfill\lthtmlcheckvsize\clearpage} + +{\newpage\clearpage +\lthtmldisplayA{displaymath170}% +\begin{displaymath}a(k+1,l+1) = (a(k,l) - X_k + X_{l+1}) \bmod M \end{displaymath}% +\lthtmldisplayZ +\hfill\lthtmlcheckvsize\clearpage} + +{\newpage\clearpage +\lthtmldisplayA{displaymath172}% +\begin{displaymath}b(k+1,l+1) = (b(k,l) - (l-k+1) X_k + a(k+1,l+1)) \bmod M \end{displaymath}% +\lthtmldisplayZ +\hfill\lthtmlcheckvsize\clearpage} + +\stepcounter{section} +\stepcounter{section} +\stepcounter{section} +\stepcounter{section} + +\end{document} diff --git a/rsync-web/tech_report/img1.gif b/rsync-web/tech_report/img1.gif new file mode 100644 index 00000000..d475550e Binary files /dev/null and b/rsync-web/tech_report/img1.gif differ diff --git a/rsync-web/tech_report/img2.gif b/rsync-web/tech_report/img2.gif new file mode 100644 index 00000000..fdc79866 Binary files /dev/null and b/rsync-web/tech_report/img2.gif differ diff --git a/rsync-web/tech_report/img3.gif b/rsync-web/tech_report/img3.gif new file mode 100644 index 00000000..3479fed6 Binary files /dev/null and b/rsync-web/tech_report/img3.gif differ diff --git a/rsync-web/tech_report/img4.gif b/rsync-web/tech_report/img4.gif new file mode 100644 index 00000000..b7ae4971 Binary files /dev/null and b/rsync-web/tech_report/img4.gif differ diff --git a/rsync-web/tech_report/img5.gif b/rsync-web/tech_report/img5.gif new file mode 100644 index 00000000..07128ec5 Binary files /dev/null and b/rsync-web/tech_report/img5.gif differ diff --git a/rsync-web/tech_report/img6.gif b/rsync-web/tech_report/img6.gif new file mode 100644 index 00000000..3ab1bf24 Binary files /dev/null and b/rsync-web/tech_report/img6.gif differ diff --git a/rsync-web/tech_report/img7.gif b/rsync-web/tech_report/img7.gif new file mode 100644 index 00000000..4a104e1a Binary files /dev/null and b/rsync-web/tech_report/img7.gif differ diff --git a/rsync-web/tech_report/index.html b/rsync-web/tech_report/index.html new file mode 100644 index 00000000..77bf9fc2 --- /dev/null +++ b/rsync-web/tech_report/index.html @@ -0,0 +1,94 @@ + + + + +The rsync algorithm + + + + + + + + + + + +next +up +previous +
    + Next: The problem +
    +
    + + +

    + +

    + +

    +

    The rsync algorithm

    +

    Andrew Tridgell         Paul Mackerras +
    +Department of Computer Science
    +Australian National University
    +Canberra, ACT 0200, Australia

    +

    + +

    + +

    Abstract:

    +
    +This report presents an algorithm for updating a file on one machine + to be identical to a file on another machine. We assume that the + two machines are connected by a low-bandwidth high-latency + bi-directional communications link. The algorithm identifies parts + of the source file which are identical to some part of the + destination file, and only sends those parts which cannot be matched + in this way. Effectively, the algorithm computes a set of + differences without having both files on the same machine. The + algorithm works best when the files are similar, but will also + function correctly and reasonably efficiently when the files are + quite different. +
    +

    +

    +


    + +  + + +

    +
    +Andrew Tridgell +
    1998-11-09 +
    + + diff --git a/rsync-web/tech_report/labels.pl b/rsync-web/tech_report/labels.pl new file mode 100644 index 00000000..eded2434 --- /dev/null +++ b/rsync-web/tech_report/labels.pl @@ -0,0 +1,6 @@ +# LaTeX2HTML 98.1p1 release (March 2nd, 1998) +# Associate labels original text with physical files. + + +1; + diff --git a/rsync-web/tech_report/next.gif b/rsync-web/tech_report/next.gif new file mode 100644 index 00000000..7a2dbe95 Binary files /dev/null and b/rsync-web/tech_report/next.gif differ diff --git a/rsync-web/tech_report/node1.html b/rsync-web/tech_report/node1.html new file mode 100644 index 00000000..83c726a4 --- /dev/null +++ b/rsync-web/tech_report/node1.html @@ -0,0 +1,119 @@ + + + + +The problem + + + + + + + + + + + + + + +next + +up + +previous +
    + Next: The rsync algorithm + Up: The rsync algorithm + Previous: The rsync algorithm +
    +
    + + +

    +The problem +

    + +

    +Imagine you have two files, A and B, and you wish to update B to be +the same as A. The obvious method is to copy A onto B. + +

    +Now imagine that the two files are on machines connected by a slow +communications link, for example a dial up IP link. If A is large, +copying A onto B will be slow. To make it faster you could +compress A before sending it, but that will usually only gain a +factor of 2 to 4. + +

    +Now assume that A and B are quite similar, perhaps both derived +from the same original file. To really speed things up you would need +to take advantage of this similarity. A common method is to send just +the differences between A and B down the link and then use this +list of differences to reconstruct the file. + +

    +The problem is that the normal methods for creating a set of +differences between two files rely on being able to read both files. +Thus they require that both files are available beforehand at one end +of the link. If they are not both available on the same machine, +these algorithms cannot be used (once you had copied the file over, +you wouldn't need the differences). This is the problem that rsync +addresses. + +

    +The rsync algorithm efficiently computes which parts of a source file +match some part of an existing destination file. These parts need not +be sent across the link; all that is needed is a reference to the part +of the destination file. Only parts of the source file which are not +matched in this way need to be sent verbatim. The receiver can then +construct a copy of the source file using the references to parts of +the existing destination file and the verbatim material. + +

    +Trivially, the data sent to the receiver can be compressed using any +of a range of common compression algorithms, for further speed +improvements. + +

    +


    + + +next + +up + +previous +
    + Next: The rsync algorithm + Up: The rsync algorithm + Previous: The rsync algorithm + +
    +Andrew Tridgell +
    1998-11-09 +
    + + diff --git a/rsync-web/tech_report/node2.html b/rsync-web/tech_report/node2.html new file mode 100644 index 00000000..c1f299d4 --- /dev/null +++ b/rsync-web/tech_report/node2.html @@ -0,0 +1,183 @@ + + + + +The rsync algorithm + + + + + + + + + + + + + + +next + +up + +previous +
    + Next: Rolling checksum + Up: The rsync algorithm + Previous: The problem +
    +
    + + +

    +The rsync algorithm +

    + +

    +Suppose we have two general purpose computers $\alpha$ +and $\beta$. +Computer $\alpha$ +has access to a file A and $\beta$ +has access to +file B, where A and B are ``similar''. There is a slow +communications link between $\alpha$ +and $\beta$. + +

    +The rsync algorithm consists of the following steps: + +

    +

    +
    1. +
    $\beta$ +splits the file B into a series of non-overlapping +fixed-sized blocks of size S bytes1. + The last block may be shorter than S bytes. + +

    +

    2. +
    For each of these blocks $\beta$ +calculates two checksums: + a weak ``rolling'' 32-bit checksum (described below) and a strong + 128-bit MD4 checksum. + +

    +

    3. +
    $\beta$ +sends these checksums to $\alpha$. + +
    4. +
    $\alpha$ +searches through A to find all blocks of length S bytes (at any offset, not just multiples of S) that have the same + weak and strong checksum as one of the blocks of B. This can be + done in a single pass very quickly using a special property of the + rolling checksum described below. + +
    5. +
    $\alpha$ +sends $\beta$ +a sequence of instructions for + constructing a copy of A. Each instruction is either a reference + to a block of B, or literal data. Literal data is sent only for + those sections of A which did not match any of the blocks of B. +
    +

    +The end result is that $\beta$ +gets a copy of A, but only the pieces +of A that are not found in B (plus a small amount of data for +checksums and block indexes) are sent over the link. The algorithm +also only requires one round trip, which minimises the impact of the +link latency. + +

    +The most important details of the algorithm are the rolling checksum +and the associated multi-alternate search mechanism which allows the +all-offsets checksum search to proceed very quickly. These will be +discussed in greater detail below. + +

    +


    + + +next + +up + +previous +
    + Next: Rolling checksum + Up: The rsync algorithm + Previous: The problem + +
    +Andrew Tridgell +
    1998-11-09 +
    + + diff --git a/rsync-web/tech_report/node3.html b/rsync-web/tech_report/node3.html new file mode 100644 index 00000000..2d36abdd --- /dev/null +++ b/rsync-web/tech_report/node3.html @@ -0,0 +1,187 @@ + + + + +Rolling checksum + + + + + + + + + + + + + + +next + +up + +previous +
    + Next: Checksum searching + Up: The rsync algorithm + Previous: The rsync algorithm +
    +
    + + +

    +Rolling checksum +

    + +

    +The weak rolling checksum used in the rsync algorithm needs to have +the property that it is very cheap to calculate the checksum of a +buffer + +X2 .. Xn+1 given the checksum of buffer + +X1 .. Xn and +the values of the bytes X1 and Xn+1. + +

    +The weak checksum algorithm we used in our implementation was inspired +by Mark Adler's adler-32 checksum. Our checksum is defined by +

    +
    + + + +\begin{displaymath}a(k,l) = (\sum_{i=k}^l X_i) \bmod M \end{displaymath} +
    +
    +

    +

    +
    + + + +\begin{displaymath}b(k,l) = (\sum_{i=k}^l (l-i+1)X_i) \bmod M \end{displaymath} +
    +
    +

    +

    +
    + + + +s(k,l) = a(k,l) + 216 b(k,l) +
    +
    +

    +

    +where s(k,l) is the rolling checksum of the bytes + +$X_k \ldots X_l$. +For simplicity and speed, we use + +M = 216. + +

    +The important property of this checksum is that successive values can +be computed very efficiently using the recurrence relations + +

    +

    +
    + + + +\begin{displaymath}a(k+1,l+1) = (a(k,l) - X_k + X_{l+1}) \bmod M \end{displaymath} +
    +
    +

    +

    +
    + + + +\begin{displaymath}b(k+1,l+1) = (b(k,l) - (l-k+1) X_k + a(k+1,l+1)) \bmod M \end{displaymath} +
    +
    +

    +

    +Thus the checksum can be calculated for blocks of length S at all +possible offsets within a file in a ``rolling'' fashion, with very +little computation at each point. + +

    +Despite its simplicity, this checksum was found to be quite adequate as +a first level check for a match of two file blocks. We have found in +practice that the probability of this checksum matching when the +blocks are not equal is quite low. This is important because the much +more expensive strong checksum must be calculated for each block where +the weak checksum matches. + +

    +


    + + +next + +up + +previous +
    + Next: Checksum searching + Up: The rsync algorithm + Previous: The rsync algorithm + +
    +Andrew Tridgell +
    1998-11-09 +
    + + diff --git a/rsync-web/tech_report/node4.html b/rsync-web/tech_report/node4.html new file mode 100644 index 00000000..de8a2723 --- /dev/null +++ b/rsync-web/tech_report/node4.html @@ -0,0 +1,145 @@ + + + + +Checksum searching + + + + + + + + + + + + + + +next + +up + +previous +
    + Next: Pipelining + Up: The rsync algorithm + Previous: Rolling checksum +
    +
    + + +

    +Checksum searching +

    + +

    +Once $\alpha$ +has received the list of checksums of the blocks of B, +it must search A for any blocks at any offset that match the +checksum of some block of B. The basic strategy is to compute the +32-bit rolling checksum for a block of length S starting at each +byte of A in turn, and for each checksum, search the list for a +match. To do this our implementation uses a +simple 3 level searching scheme. + +

    +The first level uses a 16-bit hash of the 32-bit rolling checksum and +a 216 entry hash table. The list of checksum values (i.e., the +checksums from the blocks of B) is sorted according to the 16-bit +hash of the 32-bit rolling checksum. Each entry in the hash table +points to the first element of the list for that hash value, or +contains a null value if no element of the list has that hash value. + +

    +At each offset in the file the 32-bit rolling checksum and its 16-bit +hash are calculated. If the hash table entry for that hash value is +not a null value, the second level check is invoked. + +

    +The second level check involves scanning the sorted checksum list +starting with the entry pointed to by the hash table entry, looking +for an entry whose 32-bit rolling checksum matches the current value. +The scan terminates when it reaches an entry whose 16-bit hash +differs. If this search finds a match, the third level check is +invoked. + +

    +The third level check involves calculating the strong checksum for the +current offset in the file and comparing it with the strong checksum +value in the current list entry. If the two strong checksums match, +we assume that we have found a block of A which matches a block of +B. In fact the blocks could be different, but the probability of +this is microscopic, and in practice this is a reasonable assumption. + +

    +When a match is found, $\alpha$ +sends $\beta$ +the data in A between +the current file offset and the end of the previous match, followed by +the index of the block in B that matched. This data is sent +immediately a match is found, which allows us to overlap the +communication with further computation. + +

    +If no match is found at a given offset in the file, the rolling +checksum is updated to the next offset and the search proceeds. If a +match is found, the search is restarted at the end of the matched +block. This strategy saves a considerable amount of computation for +the common case where the two files are nearly identical. In +addition, it would be a simple matter to encode the block indexes as +runs, for the common case where a portion of A matches a series of +blocks of B in order. + +

    +


    + + +next + +up + +previous +
    + Next: Pipelining + Up: The rsync algorithm + Previous: Rolling checksum + +
    +Andrew Tridgell +
    1998-11-09 +
    + + diff --git a/rsync-web/tech_report/node5.html b/rsync-web/tech_report/node5.html new file mode 100644 index 00000000..5392a93a --- /dev/null +++ b/rsync-web/tech_report/node5.html @@ -0,0 +1,85 @@ + + + + +Pipelining + + + + + + + + + + + + + + +next + +up + +previous +
    + Next: Results + Up: The rsync algorithm + Previous: Checksum searching +
    +
    + + +

    +Pipelining +

    + +

    +The above sections describe the process for constructing a copy of one +file on a remote system. If we have a several files to copy, we can +gain a considerable latency advantage by pipelining the process. + +

    +This involves $\beta$ +initiating two independent processes. One of the +processes generates and sends the checksums to $\alpha$ +while the +other receives the difference information from $\alpha$ +and +reconstructs the files. + +

    +If the communications link is buffered then these two processes can +proceed independently and the link should be kept fully utilised in +both directions for most of the time. + +

    +


    +
    +Andrew Tridgell +
    1998-11-09 +
    + + diff --git a/rsync-web/tech_report/node6.html b/rsync-web/tech_report/node6.html new file mode 100644 index 00000000..baae473e --- /dev/null +++ b/rsync-web/tech_report/node6.html @@ -0,0 +1,299 @@ + + + + +Results + + + + + + + + + + + + + + +next + +up + +previous +
    + Next: Availability + Up: The rsync algorithm + Previous: Pipelining +
    +
    + + +

    +Results +

    + +

    +To test the algorithm, tar files were created of the Linux kernel +sources for two versions of the kernel. The two kernel versions were +1.99.10 and 2.0.0. These tar files are approximately 24MB in size and +are separated by 5 released patch levels. + +

    +Out of the 2441 files in the 1.99.10 release, 291 files had changed in +the 2.0.0 release, 19 files had been removed and 25 files had been +added. + +

    +A ``diff'' of the two tar files using the standard GNU diff utility +produced over 32 thousand lines of output totalling 2.1 MB. + +

    +The following table shows the results for rsync between the two files +with a varying block size.2 + +

    +
    +
    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    block matches tag false data written read
    size  hits alarms   

    +300

    642473817434948531220056291581632284
    500469896200136410919001283906979384
    700332555719702213078001444346699564
    900256865250582414695001575438544124
    1100208484968442116545001740838445204
    +
    +
    + +

    +In each case, the CPU time taken was less than the +time it takes to run ``diff'' on the two files.3 + +

    +The columns in the table are as follows: + +

    +

    +
    block size +
    The size in bytes of the checksummed blocks. +
    matches +
    The number of times a block of B was found in A. +
    tag hits +
    The number of times the 16 bit hash of the rolling +checksum matched a hash of one of the checksums from B. +
    false alarms +
    The number of times the 32 bit rolling checksum + matched but the strong checksum didn't. +
    data +
    The amount of file data transferred verbatim, in bytes. +
    written +
    The total number of bytes written by $\alpha$ +including protocol overheads. This is almost all file data. +
    read +
    The total number of bytes read by $\alpha$ +including + protocol overheads. This is almost all checksum information. +
    +

    +The results demonstrate that for block sizes above 300 bytes, only a +small fraction (around 5%) of the file was transferred. The amount +transferred was also considerably less than the size of the diff file +that would have been transferred if the diff/patch method of updating +a remote file was used. + +

    +The checksums themselves took up a considerable amount of space, +although much less than the size of the data transferred in each +case. Each pair of checksums consumes 20 bytes: 4 bytes for the +rolling checksum plus 16 bytes for the 128-bit MD4 checksum. + +

    +The number of false alarms was less than 1/1000 of the number of +true matches, indicating that the 32 bit rolling checksum is quite +good at screening out false matches. + +

    +The number of tag hits indicates that the second level of the +checksum search algorithm was invoked about once every 50 +characters. This is quite high because the total number of blocks in +the file is a large fraction of the size of the tag hash table. For +smaller files we would expect the tag hit rate to be much closer to +the number of matches. For extremely large files, we should probably +increase the size of the hash table. + +

    +The next table shows similar results for a much smaller set of files. +In this case the files were not packed into a tar file first. Rather, +rsync was invoked with an option to recursively descend the directory +tree. The files used were from two source releases of another software +package called Samba. The total source code size is 1.7 MB and the +diff between the two releases is 4155 lines long totalling 120 kB. + +

    +
    +
    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    block matches tag false data written read
    size  hits alarms   

    +300

    37273899012977515399983948
    50021582325017157418933050908
    70015171649019502421014436828
    90011561281022284723647129048
    11009211049025007326272523988
    +
    +
    + +

    +


    + + +next + +up + +previous +
    + Next: Availability + Up: The rsync algorithm + Previous: Pipelining + +
    +Andrew Tridgell +
    1998-11-09 +
    + + diff --git a/rsync-web/tech_report/node7.html b/rsync-web/tech_report/node7.html new file mode 100644 index 00000000..1f44ad46 --- /dev/null +++ b/rsync-web/tech_report/node7.html @@ -0,0 +1,62 @@ + + + + +Availability + + + + + + + + + + + + + + +next + +up + +previous +
    + Next: About this document ... + Up: The rsync algorithm + Previous: Results +
    +
    + + +

    +Availability +

    + +

    +An implementation of rsync which provides a convenient interface +similar to the common UNIX command rcp has been written and is +available for download from ftp://rsync.samba.org/pub/rsync. + +

    +


    +
    +Andrew Tridgell +
    1998-11-09 +
    + + diff --git a/rsync-web/tech_report/node8.html b/rsync-web/tech_report/node8.html new file mode 100644 index 00000000..0bf88a51 --- /dev/null +++ b/rsync-web/tech_report/node8.html @@ -0,0 +1,61 @@ + + + + +About this document ... + + + + + + + + + + + +next + +up + +previous +
    + Up: The rsync algorithm + Previous: Availability +
    +
    + + +

    + About this document ... +

    + The rsync algorithm

    +This document was generated using the +LaTeX2HTML translator Version 98.1p1 release (March 2nd, 1998) +

    +Copyright © 1993, 1994, 1995, 1996, 1997, +Nikos Drakos, +Computer Based Learning Unit, University of Leeds. +

    +The command line arguments were:
    + latex2html tech_report.tex. +

    +The translation was initiated by Andrew Tridgell on 1998-11-09 +


    +
    +Andrew Tridgell +
    1998-11-09 +
    + + diff --git a/rsync-web/tech_report/previous.gif b/rsync-web/tech_report/previous.gif new file mode 100644 index 00000000..aef90f1f Binary files /dev/null and b/rsync-web/tech_report/previous.gif differ diff --git a/rsync-web/tech_report/tech_report.css b/rsync-web/tech_report/tech_report.css new file mode 100644 index 00000000..4f86ccfc --- /dev/null +++ b/rsync-web/tech_report/tech_report.css @@ -0,0 +1,19 @@ + +/* Century Schoolbook font is very similar to Computer Modern Math: cmmi */ +.MATH { font-family: "Century Schoolbook", serif; } +.MATH I { font-family: "Century Schoolbook", serif; font-weight: bold } +.BOLDMATH { font-family: "Century Schoolbook", serif; font-weight: bold } + +/* implement both fixed-size and relative sizes */ +SMALL.XTINY { font-size : xx-small } +SMALL.TINY { font-size : x-small } +SMALL.SCRIPTSIZE { font-size : smaller } +SMALL.FOOTNOTESIZE { font-size : small } +SMALL.SMALL { } +BIG.LARGE { } +BIG.XLARGE { font-size : large } +BIG.XXLARGE { font-size : x-large } +BIG.HUGE { font-size : larger } +BIG.XHUGE { font-size : xx-large } + +/* document-specific styles come next */ diff --git a/rsync-web/tech_report/tech_report.html b/rsync-web/tech_report/tech_report.html new file mode 100644 index 00000000..77bf9fc2 --- /dev/null +++ b/rsync-web/tech_report/tech_report.html @@ -0,0 +1,94 @@ + + + + +The rsync algorithm + + + + + + + + + + + +next +up +previous +
    + Next: The problem +
    +
    + + +

    + +

    + +

    +

    The rsync algorithm

    +

    Andrew Tridgell         Paul Mackerras +
    +Department of Computer Science
    +Australian National University
    +Canberra, ACT 0200, Australia

    +

    + +

    + +

    Abstract:

    +
    +This report presents an algorithm for updating a file on one machine + to be identical to a file on another machine. We assume that the + two machines are connected by a low-bandwidth high-latency + bi-directional communications link. The algorithm identifies parts + of the source file which are identical to some part of the + destination file, and only sends those parts which cannot be matched + in this way. Effectively, the algorithm computes a set of + differences without having both files on the same machine. The + algorithm works best when the files are similar, but will also + function correctly and reasonably efficiently when the files are + quite different. +
    +

    +

    +


    + +  + + +

    +
    +Andrew Tridgell +
    1998-11-09 +
    + + diff --git a/rsync-web/tech_report/up.gif b/rsync-web/tech_report/up.gif new file mode 100644 index 00000000..3d1aebf3 Binary files /dev/null and b/rsync-web/tech_report/up.gif differ diff --git a/rsync-web/vger.txt b/rsync-web/vger.txt new file mode 100644 index 00000000..da629c9a --- /dev/null +++ b/rsync-web/vger.txt @@ -0,0 +1,35 @@ +The vger.rutgers.edu cvs tree is mirrored onto cvs.samba.org via +anonymous rsync using the following script. + + #!/bin/bash + + cd /var/www/cvs/vger/ + PATH=/usr/local/bin:/usr/freeware/bin:/usr/bin:/bin + + RUN=`lps x | grep rsync | grep -v grep | wc -l` + if [ "$RUN" -gt 0 ]; then + echo already running + exit 1 + fi + + rsync -az vger.rutgers.edu::cvs/CVSROOT/ChangeLog $HOME/ChangeLog + + sum1=`sum $HOME/ChangeLog` + sum2=`sum /var/www/cvs/vger/CVSROOT/ChangeLog` + + if [ "$sum1" = "$sum2" ]; then + echo nothing to do + exit 0 + fi + + rsync -az --delete --force vger.rutgers.edu::cvs/ /var/www/cvs/vger/ + exit 0 + +Note in particular the initial rsync of the ChangeLog to determine if +anything has changed. This could be omitted but it would mean that the +rsyncd on vger would have to build a complete listing of the cvs area +at each run. As most of the time nothing will have changed I wanted to +save the time on vger by only doing a full rsync if the ChangeLog has +changed. This helped quite a lot because vger is low on memory and +generally quite heavily loaded, so doing a listing on such a large +tree every hour would have been excessive. diff --git a/rsync-web/win95.txt b/rsync-web/win95.txt new file mode 100644 index 00000000..d856682f --- /dev/null +++ b/rsync-web/win95.txt @@ -0,0 +1,110 @@ +Compiling rsync for MS Windows 95 +================================= + +Caveat: +There are no doubt other ways to do this, but I have not tried them. + +This will produce an rsync binary which runs under win95/NT which can _send_ +data to a UNIX rsync host but not receive. This is suitable for my requirements +as I am backing up a bunch of win95 machines to a linux server, but it may +not be what _you_ require. + +NOTE: I am now told (by patl@curl.com) that receiving as well as +sending works with the current cygwin32 toolkit. + + +Step 1. Get the Cygnus development kit +-------------------------------------- + +If you do not already know about this wonderful project undertaken by cygnus, +grab the latest copy of the gnu-win32 runtime system from + + http://www.cygnus.com/misc/gnu-win32/ + +You will want the CDK-B19.EXE self-installing binary (be warned - it's 12M +in size) which contains gcc+binutils+DLLs required by the system. Read the +instructions about this and install it. You will then have something which +looks _very_ close to a UNIX box as far as any UNIX software ic concerned. + +Also grab the updated version (B19.1) of the DLL as this fixes some problems +(it's pretty small at ~190k). + +Take a bit of time to play with this stuff - have a look at the registry +entries that define mount points etc so you know where/how your windows +directories are mapped. + +Step 2. Compile rsync +--------------------- + +Once the magic BASH$ prompt appears in your cygnus dos box, you can download +and compile rsync right out of the box. You might want to do this: + + tar xvzf rsync-2.0.17.tar.gz (or whatever version...) + cd rsync-2.0.17 + chmod +x configure (exec is not set automatically) + ./configure + make + +and you will have a working RSYNC.EXE + +Before you can do a "make install" you must edit the Makefile so that +it knows about "rsync.exe" rather than "rsync". + +Step 3 (optional) - compile and install SSH +------------------------------------------- + +If you want to use ssh as the transport then you better grab a copy and compile +it - again, no surprises. It compiles ok under gnuwin32, and you can generate +keys etc in the normal way. + +Problems +-------- + +Well, it _almost_ works seamlessly. Here are a couple of problems I have +encountered so far - I do not know why these symptoms occur, and they are +probably nothing to do with rsync, but if you follow this course then you +will probably run into these things also... + +1. rsync will not _receive_ data onto a windows box. It seems to do everything + ok, but then hangs at the last step and no files actually appear. When + transferring via my modem, I can see data being received, but it never + appears in my directory. + +NOTE: This now appears to be fixed. + +2. rsync, ssh and other gnuwin32-compiled programs can be called directly from + a DOS prompt, but argument passing seems to be broken. This means that + starting SSH directly is difficult. + + After a bit of experimenting I found a workaround, which is not pretty + but worked for me :-) + + Lets say you want to call ssh from a normal DOS prompt. Set up a batch + file called "SSH.BAT" which sets up the cygnus environment and calls bash + with the argument "ssh.sh %1 %2 %3 %4 %5 %6 %7 %8 %9". You can copy the + cygnus.bat file and use it as a template if you like. here is my ssh.bat: + + + @ECHO OFF + SET TERM=ansi + SET MAKE_MODE=unix + SET CYGFS=e:/apps/cygnus + SET GCC_EXEC_PREFIX=/lib\gcc-lib\ + SET TCL_LIBRARY=%CYGROOT%\share\tcl8.0\ + SET HOME=/home/awesley + SET GDBTK_LIBRARY=%CYGFS%/share/gdbtcl + bash /scripts/ssh.sh %1 %2 %3 %4 %5 %6 %7 %8 %9 + + + Create a shell script called ssh.sh containing: + + /usr/local/bin/ssh $* + + which is then run by bash and calls the ssh binary (wherever you have + installed it). + + I suspect that there is a better way, but I have not yet had time to + look for it. + + +Cheers, Anthony Wesley diff --git a/rsync-web/y2k.html b/rsync-web/y2k.html new file mode 100644 index 00000000..90cc7625 --- /dev/null +++ b/rsync-web/y2k.html @@ -0,0 +1,24 @@ + + + +rsync y2k issues + + + +

    Year 2000 Issues

    + +We are starting to get a lot of Y2K compliance questions. The answer +is that rsync has no Y2K problems. rsync uses 32 bit "seconds since +1970" date formats and thus has no problems with the turn of the +century.

    + +You must remember though that rsync is part of a larger system. It +relies on a transport mechanism (rsh, ssh etc) and many operating +systems services (such as inetd, syslog etc). If any of these other +parts fail then rsync will of course be affected.

    + +Also remember that rsync comes with absolutely no warranty. It is up +to you to test whether rsync is suitable in your computing environment, +both before and after the turn of the century. + +