diff options
20 files changed, 690 insertions, 4012 deletions
diff --git a/electra-background.txt b/electra-background.txt new file mode 100644 index 0000000..7201c95 --- /dev/null +++ b/electra-background.txt @@ -0,0 +1,478 @@ +# Training In Sailing Vessels Carries On + +By Commander Francis E. Clark, U. S. Navy +[View +Issue](https://www.usni.org/magazines/proceedings/1955/october){.btn + +Training in seamanship under sail for young men is not dead, but is +still very much alive! We are aware that the age of nuclear power has +arrived and may, in the not too distant future, replace steam, but how +many realize that sailing vessels, even in our own country, provide +basic training which certain people believe cannot be adequately +duplicated in full-powered ships? + +It has been over forty years since the U. S. Navy has provided formal +training in square-rigged sailing vessels for midshipmen and recruit +seamen, and there are probably very few, if any, officers on active duty +today whose personal memories include going aloft and laying out on a +yardarm. We, in the Navy, have lost direct contact with the "old" days +of sail for so long that it may be somewhat of a surprise that sail +training is still firmly believed in and *supported* by many of the +maritime nations of the world. USCGC *Eagle* (fully pictured in the +[October, 1954, +*Proceedings*](https://www.usni.org/magazines/proceedings/1954/october/eagle-spreads-her-wings-pictorial)) +is not by any means an isolated example; in this year of our Lord 1955, +there are at least *fifteen* square-riggers engaged in training, all +operational and seagoing. There are a dozen or so more whose present +status is not known, mainly because they belong to countries behind the +Iron Curtain, but which are possibly operational. By "square-riggers" is +meant vessels rigged as ships, barks, and barkentines; in addition there +are a number of small sail-craft in service, topsail schooners, ketches, +and so on. Most, of course, have auxiliary power. + +Although World War II appreciably reduced the number of these craft, the +survivors are not merely dying relics of a past era. Interest in this +type of training is continuing and is reflected by the number of sailing +vessels which were refitted after wartime damage and by new +construction. Since World War II, at least five (and possibly quite a +few more) *new* sail training vessels have been completed---several +barkentines and schooners, and one brigantine. Furthermore, most of the +older ones are receiving excellent material upkeep, including extensive +refits when necessary. + +This writer does not intend to enter the controversy as to whether sail +training (as opposed to training on a vessel with only mechanical means +of propulsion) is necessary or desirable. Suffice it to say that +numerous persons in positions of responsibility and authority do believe +in sail training. The subject has been well aired by many writers and by +men of experience for years. Some of the remarks concerning *Eagle* in +the [October, 1954, +*Proceedings*](https://www.usni.org/magazines/proceedings/1954/october/eagle-spreads-her-wings-pictorial) +contain the essential reasons in favor of sail training; another example +is an article in the [September, 1938, +*Proceedings*](https://www.usni.org/magazines/proceedings/1938/september/why-training-sail). +No advocate attempts to argue that sail training is required to teach +the midshipman or recruit professional subjects; in fact, it is admitted +that in many fields, engineering, for example, a "modern" training +vessel is preferable from a technical standpoint. The proponents of sail +training place their emphasis on the more intangible benefits of +character building induced by life under sail. These benefits are +considered so valuable that sail training is still supported, not only +by governments, but also by hard- headed businessmen. For example, +Norway today has three sail training vessels supported primarily, to the +best of my knowledge, by private funds (merchant shipowners, *etc*.); +yet the last commercial square- rigger under the Norwegian flag (the +bark *Peltr Ugland)* made its last voyage in 1929. Certainly these three +vessels are not maintained to teach young men the technical details of +handling a ship under sail. + +It may also be noted, as will be seen below, that facilities for sail +training are being increased in certain countries, although dying out in +others. Our own United States is one example; at least the U. S. Coast +Guard cadets are receiving more extensive sail training in the *Eagle* +than they did aboard the pre-World War II *Alexander Hamilton.* Brazil +now has two large sailers instead of one. Indonesia, a new nation, has +the newly constructed *Dewarutji.* And the USSR, instead of one large +sail training vessel, now has a sizable fleet! + +Before attempting to summarize recent and present sail training vessels, +mention should be made of the various types of training ships from the +standpoint of sponsorship, although the line of demarcation is not +always clear. First of all are those entirely government owned. Naval +training vessels are obviously in this category, used for the training +of midshipmen (or prospective officers by whatever term they are +designated), or enlisted personnel, or both. These functions are carried +out either in separate training vessels or sometimes together. *Eagle,* +for example, carries only cadets (prospective officers); the Portuguese +*Sagres* carries cadets, enlisted recruits, and also trainees of varying +grades preparing for advancement in rating, a program similar to our +class "B" schools. Most of the naval training vessels are associated +with a shore facility such as a naval academy. Second, there are +government training vessels primarily in connection with merchant marine +training, but some may also train naval cadets. These are similar to the +naval training ships and may or may not be associated with a shore +facility. Boys of different ages may be separately trained; for example, +the Danish *Georg Stage* trains younger boys on Baltic and North Sea +cruises, and many, although not all, then "graduate" to deep sea +training in the *Danmark.* The third type is primarily privately +sponsored, although there may be a government subsidy. These may be +operated by a large shipping company (primarily for its own benefit), by +an association of shipowners in a particular port, or by a "foundation," +usually endowed originally by a wealthy man with an interest in shipping +and in the training of young men. Some of these foundations may be +considered charities in that applicants are limited to certain income +brackets. In this article, I use the words "private" or "foundation" to +designate these types of sponsorship; merchant marine training will be +understood. + +Other forms of classification are whether the vessel is stationary or +operational, and whether the vessel does or does not carry cargoes as a +secondary mission. Obviously this latter class applies primarily to +those privately sponsored. The combined cargotraining scheme is favored +by large shipping companies, both to reduce the cost and to provide +training in cargo handling. + +Another form of training (merchant marine), which is outside the scope +of this article, is the employment of cadets or apprentices in regular +commercial vessels as a prerequisite to "sitting" for an officer's +license. This form of training is well known and many countries, at +least before World War II, required varying amounts of this +apprenticeship to be served in sail. Therefore, when sailing vessels +were becoming scarcer, many firms had vessels which were designated +"cadet ships" and carried in their crews a high percentage of "boys" who +required this experience and were willing to receive minimum wages, or +even *pay* a premium for the privilege of so serving. The British firm +of Devitt and Moore in pre-World War I days, the German nitrate carriers +belonging to Herr Laeisz, and many of Gustav Erikson's fleet (Finnish) +in the years between wars were especially notable in this regard but +cannot be further described here. However, for these, profitable +operation was essential to their existence, and training was necessarily +a subordinate function, but the distinction between "cadet ships" and +privately sponsored cargo-training vessels is often very slight. + +With this background, a summary of the sail training vessels of the +present and recent past is in order. I have grouped them roughly by +geographical area, for convenience; my information is obtained from +Jane's *Fighting Ships,* Lloyd's *Register,* and various books and +publications dealing with maritime history. Starting with our own +country, the U. S. Navy has had no formal sail training for many years; +USS *Hartford* in 1909 was the last sailing vessel to make a +midshipmen's practice cruise, and when the *Reina Mercedes* replaced the +USS *Hartford* as station ship in 1912, square rig disappeared forever +from the Naval Academy. (An excellent history of the Naval Academy +practice ships will be found in the May, 1934, Proceedings.) Since then +the only sail training, except for small boats, at the U. S. Naval +Academy has been on a voluntary extracurricular basis on board attached +yachts. Older officers will remember the *Argo* and *Robert Center;* +those more junior will be familiar with *Vamarie, Highland Light, +Freedom, Royono,* and the Luders yawls presently stationed at Annapolis. +Our Coast Guard, however, is still an adherent of sail training for its +prospective officers. In earlier days, the *Dobbin, Chase,* and *Itasca* +were successively the practice ships, and, until 1900, constituted the +whole Coast Guard Academy. In 1920 the gunboat *Vicksburg* (PG 11), an +auxiliary barkentine, was acquired from the Navy and renamed *Alexander +Hamilton,* and was based at New London until the late thirties. (There +were six gunboats in this 1895 class which included *Newport* (PG 12) +and *Annapolis* (PG 10), to be mentioned later.) During World War II, +the Danish *Danmark,* fortunately in this country when her homeland was +occupied, was volunteered by her master, and ship and crew served the +Coast Guard Academy throughout the war, making short cruises under sail +in Long Island Sound and nearby waters. Since then, the auxiliary bark +*Eagle* (the former German *Horst Wessel*) was acquired and needs no +further comment here. Another sailer in Coast Guard service from 1941 to +1947 was the three-masted schooner yacht *Atlantic,* which in 1905 won a +trans-Atlantic race with a passage of twelve days, four hours, from +Sandy Hook to the Lizard. + +The United States government has consistently supported \"merchant +marine training by the loan of suitable vessels from the Navy to the +various State Maritime Academies; the states of New York, Pennsylvania +and Massachusetts formerly had sail training vessels. The old sloop of +war *St. Marys* served New York until replaced by the *Newport* about +1907. *Newport* (which participated in the Naval Academy cruise of 1900) +was normally berthed at Bedloe's Island in New York Harbor (this was +prior to the acquisition of the Fort Schuyler site), and made her last +annual cruise in 1931 to Bremerton Navy Yard, where USS *Procyon* +(steam) replaced her. A sister ship, *Annapolis* (which participated in +the Naval Academy practice cruise of 1899), replaced *Saratoga* as the +Pennsylvania schoolship and served until shortly before World War II. +Massachusetts' first training ship was the USS *Enterprise,* loaned in +1892 and relieved in 1909 by the USS *Ranger* (PG 23), renamed the +*Nantucket* in 1918. *Ranger* was built in 1873-1876 at Wilmington, +Delaware, as an auxiliary bark, most of her naval career being on survey +and patrol duties in the Pacific. She was rerigged as a barkentine at +Mare Island in 1896-99, but the yards on her mainmast were replaced +during an extensive overhaul at Boston Navy Yard in 1932, and she +continued active until World War II threatened. All the other state +training vessels (including the present ones) were, and are, +full-powered. + +The U. S. government took a direct hand in merchant marine training in +World War II. *Nantucket,* briefly renamed *Bay State,* was transferred +to the newly founded U. S. Merchant Marine Academy at King's Point N. Y. +in 1942 and again renamed, this time being christened *Emery Rice.* +Operational until 1944, she is now designated as a museum ship. The +three-masted auxiliary schooner (yacht) *Verna* was also at King's Point +at one time, but is there no longer. For the wartime school at St. +Petersburg, Florida, the Maritime Commission obtained the ship +*Tusitala* (built 1883 at Greenock, Scotland, as the British merchantman +*Inveruglas,* later *Sierra Lucena, Sofie,* and *Tusitala)* as a +stationary training ship, and also the little full-rigged ship *Joseph +Conrad,* built in Denmark in 1882 as the *Georg Stage. Joseph Conrad* +was employed under sail out of St. Petersburg until 1945. She is still +afloat, although not operational, and is owned by the Marine Historical +Association, Inc., of Mystic, Connecticut. The *Conrad* is used by Sea +Scout groups. *Tusitala* was scrapped in 1948 at Mobile, Alabama. + +Among privately sponsored U. S. vessels, mention should be made of the +five-masted barkentine *Marsala,* operated in the 1930's by the American +Nautical Academy National Training School of Washington, D. C. Tabor +Academy, in Marion, Massachusetts, a boy's preparatory school and a +Navy- designated "honor" school, has had several yachts, usually named +*Tabor Boy.* The present one was built about 1915 in Amsterdam as "Pilot +Schooner No. 2" (there were twelve in all) and in use as such until +1926. Later she was renamed *Bestevaer* and used for training, at first +out of Rotterdam, and after 1943 by the German Navy in the Baltic. After +a brief sojourn in Soviet hands, *Bestevaer* was returned to the +Netherlands and in 1953 was purchased by Mr. R. C. Allen of Grand +Rapids, Michigan, and later presented to Tabor Academy, where she is now +in active service. Finally, in 1953 the Catholic Sea Cadets of America +acquired the four- masted schooner *Annie C. Ross* (built 1917). Renamed +Star of the Sea, she quietly sank to the bottom near Hempstead, New +York, on September 4, 1955. + +The other American republics can be covered more quickly, and most of +the Latin American training ships are under naval sponsorship. Well +known is the Brazilian Navy's *Almirante Saldanha,* an auxiliary +four-masted barkentine which has frequently visited the United States. +This is the second vessel of that name, the first being a full- rigged +ship, the former British merchantman *Dovenby Hall* (built 1885) which +served Brazil's merchant marine during the twenties. The current +*Almirante Saldanha* was launched at Barrow, England, December 19, 1933. +In 1948 Brazil received a second seagoing naval training ship, the +auxiliary bark *Guanabara,* built in 1937 as the German *Albert Leo +Schlageter,* and a sister ship to USCGC *Eagle.* I emphasize the word +*second* since *Almirante Saldanha* received a major refit at her +builder's yard in 1950. To further emphasize Brazil's current interest +in sail training, the small schooner *Albatros* (formerly *Wishbone)* +was acquired from England about 1950 and attached to the Naval College. +The Argentine Navy has long had the auxiliary ship *Presidente +Sarmiento,* built at Birkenhead in 1898, refitted in the same yard in +1925-26, and again refitted 1940-41. She has also served as the +presidential yacht. Chile's *General Baquedano* was similar but +bark-rigged. She was also built in England in 1898 and extensively +refitted in the twenties, but has not been sea-going since before World +War II, and is now a hulk, no longer listed in *Jane\'s.* However, in +1941, Chile acquired the four-masted bark *Priwall* from Germany, one of +the finest of the Laeisz "Flying P" nitrate carriers. *Priwall* was +commissioned in the Chilean Navy as *Lautaro*, later fitted with +auxiliary diesels, and made several combined cargo-training runs to +California ---a rare example of a *naval* cargo-carrying training +ship---until she was lost by fire off the coast of Peru in March, 1945. +The present Chilean training ship is the *Esmeralda,* newly constructed +at Cadiz, Spain, and launched as recently as May 12, 1953. An auxiliary +barkentine (See page 1186), she was originally intended for the Spanish +Navy and is reported to be similar to *Juan Sebastian de Elcano.* +However, she was never commissioned in the Spanish Navy but transferred +directly to Chile upon completion. To complete the Latin American +roster, Peru had the four-masted bark *Contramaestre Duenas* (ex-British +*Vortigern)* for a period subsequent to World War I. Uruguay formerly +had the seagoing schooner *Aspirante* and the stationary barkentine +*Diez y Ocho de Julio,* but both of these were scrapped in 1953. +Colombia has the ketch *La Atrevida,* and the Dominican Republic has +*Duarte,* listed in *Jane\'s.* + +The Pacific area can also be covered briefly. Australia and New Zealand +have not had underway sail training since World War I. Japan, however, +had a quartet of auxiliary four-masted sailing vessels for merchant +marine training, the barks *Taisei Maru* (1904), *Nippon Maru* (1930), +*Kaio Maru* (1930), and the barkentine *Shintoku Maru* (1924). The best +known was probably *Taisei Maru,* of the Tokyo Nautical College, which +several times visited the United States and once circumnavigated the +world. In 1929, for example, she commenced her fortieth training +voyage---to various Pacific Islands, Formosa, Hong Kong, and Manila, +returning to Tokyo in April, 1930. The sisters *Nippon Maru* and *Kaio +Maru* (as spelled by Lloyd's but also sometimes seen as *Kaiwo* or +*Kaimo Maru)* were built at Kobe and operated from Tokyo; the writer saw +them at Tsingtao in 1940. All three survived World War II, being +stripped of their sails and used as motor vessels; *Nippon Maru,* +however, was rerigged in 1952 and, again operational, visited Portland, +Oregon, in 1955. *Shintoku Maru* was formerly attached to the Kobe +Nautical College, but disappeared from Lloyd's *Register* in the +thirties. A newcomer in the Pacific is the Indonesian auxiliary +barkentine *Dewarutji,* launched at Hamburg on January 24,1953. Veteran +German school- ship personnel assisted the Indonesians with her trials, +shakedown, and delivery via Suez Canal. + +Europe has been, and is, the stronghold of sail training, However, +rather surprisingly, Great Britain is not the leading European country +in this regard. For the Royal Navy, the only recent name I know is the +yawl *Amaryllis,* formerly attached to the Royal Naval College at +Dartmouth. For the far- flung merchant marine on which Britannia +depends, I know of no large sea-going vessels since Devitt and Moore's +"cadet ships" were dispersed by World War I, except the round-the-world +cruise of *Joseph Conrad* in 1934---36. Although square-rig experience +was required until recently for certain licences, such as Thames River +Pilot, there was no vessel under the British flag where this experience +could be obtained after the *Walerwilch,* a small coastal barkentine, +was sold in 1939! However, the British government has in the past +supported the stationary training ships *Conway, Arelhusa,* and +*Worcester,* at least to the extent of providing aging men-of-war for +the purpose, although these schools are better described as "private" +rather than "government." The latest *Conway* was originally the +ship-of-the-line HMS *Nile,* laid down in 1827, but not initially +commissioned until 1852; moored in the Mersey as the third *Conway* +since about 1875, she was taken to Menai Strait for safety during World +War II. In 1953 while being towed through the Swillies en route to +Birkenhead for drydocking, the towline parted and she went aground, +breaking her back. The school will carry on ashore. *Worcester* was also +originally one of England's "wooden walls" but for many years was moored +near London as part of the Thames Nautical Training College. She was +joined in 1938 by the former British clipper ship *Cutty Sark.* The +"old" *Worcester* has now gone, but a new *Worcester* training hulk was +recently fabricated to replace her; *Cutty Sark* is being preserved as a +relic. The Shaftesbury Home and Arethusa Training Ship, near London, is +similar. The original *Arethusa,* an 1849 frigate reportedly the last +Royal Navy vessel to go into action under sail, was moored in the Medway +for sixty years until scrapped in 1933. Her replacement was the +four-masted bark *Peking,* another smart Laeisz "Flying P" vessel, which +is still afloat (now crossing yards only on the fore) and, of course, +renamed *Arethusa.* Operational British sail training vessels are all +small, privately sponsored, and largely former yachts. Between wars they +included the barkentines *St. George* and *Lady Quirk* and the schooners +*Maisie Graham* and *Exmouth II* and, since\' World War II, the +*Warspite, Moyanna,* and *Garibaldi.* A 1955 addition is the former +Danish three-masted schooner *Peder Most,* now being refitted and +renamed *Prince Louis II.* The Antarctic exploration bark *Discovery,* +used by Sea Scouts for many years, is still moored in the Thames. + +France is largely uninterested. *Jane\'s* lists several small training +vessels for the Navy, but the largest are the 227-ton sisters *L' +Etoile* and *LaBelle-Poule,* topsail schooners built at Fecamp in 1932. +For the merchant marine, the four-masted bark *Richelieu* (former German +*Pola*) was placed in service after World War I, but she sank after a +cargo explosion at Baltimore in 1926. Since then the bark *Charles +Danielou* was bought in 1930 but quickly faded from the scene. The +Netherlands are more interested. The Royal Netherlands Navy has the +small schooner *Urania,* commissioned in 1938, and *Hobein,* a +post-World War II acquisition (ex-German). For the merchant marine there +are the stationary school ships *Nederlander* at Rotterdam, and *Pollux* +(bark) at Amsterdam. *Bestevaer,* previously mentioned, was privately +sponsored prior to World War II. And Belgium has been a steady provider +of sail training for the merchant marine. In 1906 the ship *Comte de +Smet de Naeyer* (built 1877 as the British *Jeanie Landles,* but best +known as *Linlithgowshire)* was acquired and used as a stationary +training ship at Antwerp until she was scrapped in 1934. Seagoing +training was provided by the four-masted bark *L'Avenir,* built +specially in Germany in 1908. *L'Avenir's* primary mission was training, +but she also carried cargoes, her last one for the Belgians being +phosphate from Fernandina, Florida, in 1932. Both of these were replaced +by *Mercator; L'Avenir* was sold to Gustav Erikson of Finland and will +be mentioned again. *Mercator* is a three-masted auxiliary barken- tine, +built for Belgium at Leith, Scotland, in 1932, specifically for training +purposes. She cruises extensively in the Atlantic each year and at least +once in the Pacific. World War II found her en route from Rio to St. +Helena, but she made her way safely to Freetown. *Mercator* visited New +York in 1953, and Philadelphia and Boston in 1954. Under private +sponsorship was the ketch *Strombank,* in the thirties, at Zeebrugge. + +For a nation as devoted to sail training as Germany, it is unfortunate +that she has had such bad luck. Twice in this century she lost her best +training ships as "reparations." And she has had most of the tragedies +for this class of vessel: *Niobe* foundered, *Bohus* and *Pommern* were +lost in heavy weather, and *Admiral Karpfanger* went missing with all +hands. The republic of West Germany is still interested but presently +has no government sponsored deep-sea sail training vessel. *Deutschland* +is still afloat but laid up in poor condition; *Seute Deem* is now a +youth hostel ship in the Netherlands. One post-war attempt by private +capital ran into economic difficulties; the four-masted barks *Pamir* +and *Passat,* whose last grain voyages around Cape Horn were described +in the May, 1950, Proceedings, were acquired, fitted with auxiliary +power, and made two cargo-training voyages to Rio de Janeiro in 1952. +Both were then laid up, but 1955 reports are that *Pamir,* supported by +German shipowners, has resumed voyaging to South America and that +*Passat,* after overhaul, will soon join her. + +Soviet Russia, as previously mentioned, has shown a greatly increased +interest in sail training since World War II but not too many details +are available. I know of no large training vessels in the Baltic in +prewar years, but there were probably some smaller craft such as +schooners, *etc.-,* the barkentine *Vega* being one shadowy name. In the +Black Sea, however, there has usually been a larger vessel. Prior to the +Revolution, there was the *Grand Duchess Maria Nico- laevna* (ex-Devitt +and Moore "cadet ship" *Hesperus*), but she was never in Soviet hands. +However, in 1925 the U.S.S.R. acquired the former British four-masted +bark *Lauriston;* renamed *Tovarisch,* she made one or two deep-sea +voyages, sinking the Italian steamer *Alcantara* with heavy loss of +life, prior to proceeding to the Black Sea in 1928. The first +*Tovarisch* was sunk by German air action in 1944, but a second +Tovarisch (ex- German *Gorch Pock*) and the ex-Italian *Cristoforo +Colombo* (renamed ?), acquired as reparations, have replaced her. In the +Baltic, the Soviet Union reportedly now has the training ships given on +page 1147. + +Lastly, for the U.S.S.R., an auxiliary training barkentine named +*Sekstan* called at Singapore and Hong Kong in September, 1948, en route +to Vladivostok. + +Turning southward, we find Spain and Portugal both active in the sail +training field. The Portuguese Navy makes good use of the auxiliary bark +*Sagres* for many phases of training, cruising extensively. Built in +1896 as a German merchantman, she carried various names before being +purchased for naval use in 1924. Auxiliary power was fitted in 1931, but +is still little used at sea. *Sagres* will be remembered for her visits +to New York and various New England ports in 1948. There is also an old +frigate-type stationary schoolship in the Tagus named *Dom Fernando II E +Gloria.* The Spanish Navy's *Juan Sebastian de Elcano* is also well +known and has frequently visited the United States. Built in 1927-28, +she is usually called an auxiliary "barkentine," although her foremast +is not square-rigged in the traditional manner (this applies also to +some other "barkentines" so termed in this article), and some insist she +is a "topsail schooner." Her cruises are lengthy, usually in the North +and South Atlantic, but at least twice to the Pacific. For example, her +1953-54 cruise lasted 270 days, covered 21,800 miles, and included +visits to ports as distant as Cape Town, Buenos Aires, Savannah, and +Boston. As previously noted, a running mate for the *Elcano* was built +(tentatively named *Don Juan de Austria)* but was sold to Chile and is +now *Esmeralda.* This may have sparked more extensive use of the +auxiliary bark *Galatea,* built in 1896 and on the Spanish Navy list +since 1922, but long seen only in local waters; she is now in active +deep-sea service, visiting New York in December 1953. However, the small +schooners *Estella* and *Virgen de la Caridad* were "discarded" in 1953. + +Obviously, the status of sail training is quite healthy. And even where +World War II inflicted serious wounds, as it did to navies and merchant +marines throughout the world, sail training is showing a remarkable +recovery. Though commercial sailing vessels are a thing of the past with +a few very localized exceptions, the belief in the value of this form of +training is still strong. For some time to come, many youths who feel +the lure of the sea will receive their early indoctrination aboard a +vessel where the motive power is provided by the free winds, and where a +seaman's character is molded by the tall masts and tapering spars. +Although what is learned in these ships will be applied aboard vessels +with other---possibly even nuclear---forms of power, these youths will +never forget the basic respect for nature which they learned under sail. +And as long as one of these vessels remains at sea, the eyes of sailors +everywhere will appreciate their beauty and their usefulness. + +(Editor's Note: Data concerning sailing vessels used for training will +be found in tabulated form according to nationality on the following +pages.) +[Commander Francis E. Clark, U. S. +Navy](https://www.usni.org/people/francis-e-clark) +Graduated from the U. S. Naval Academy in the Class of 1937, Commander +Clark is currently attached to Headquarters, First Naval District, +Boston, Massachusetts. Since World War II he has served as Commanding +Officer of USS *Whitewood* (AG 129), USS *Redbud* (AKL 398), and USS +*Shadwell* (LSD 15). This is his first article to appear in the +Proceedings. + +[More Stories From This +Author](https://www.usni.org/people/francis-e-clark){.btn .btn-primary +.button} [View +Biography](https://www.usni.org/people/francis-e-clark){.btn +.btn-secondary .button} + +**Digital *Proceedings* content made possible by a gift from CAPT Roger +Ekman, USN (Ret.)** + +## Footer menu {#block-usni-bootstrap-footer-menu .visually-hidden} + +- [About the Naval + Institute](https://www.usni.org/about-us/mission-and-vision){drupal-link-system-path="node/30"} +- [Books & + Press](https://www.usni.org/press/books){drupal-link-system-path="press/books"} +- [Naval History](https://www.usni.org/naval-history-magazine) +- [USNI News](https://news.usni.org/) +- [Proceedings](https://www.usni.org/proceedings-magazine) +- [Oral + Histories](https://www.usni.org/press/oral-histories){drupal-link-system-path="press/oral-histories"} +- [Events](https://www.usni.org/events){drupal-link-system-path="node/73"} +- [Naval Institute + Foundation](https://www.usni.org/donate){drupal-link-system-path="node/54"} +- [Photos & Historical Prints](https://photos.usni.org/) +- [Advertise With + Us](https://www.usni.org/advertise-us "Advertise With Us") +- [Naval Institute + Archives](https://www.usni.org/archives/about){drupal-link-system-path="node/52"} + diff --git a/saved-articles/an introduction to unix.txt b/saved-articles/an introduction to unix.txt deleted file mode 100644 index 302c268..0000000 --- a/saved-articles/an introduction to unix.txt +++ /dev/null @@ -1,1913 +0,0 @@ ---- -title: Oliver | An Introduction to Unix -date: 2015-03-13T14:08:55Z -source: http://www.oliverelliott.org/article/computing/tut_unix/#100UsefulUnixCommands -tags: #cmdline - ---- - -**Everybody Knows How to Use a Computer, but Not Everyone Knows How to Use the Command Line. Yet This is the Gateway to Doing Anything and Everything Sophisticated with a Computer and the Most Natural Starting Place to Learn Programming** -_by Oliver; Jan. 13, 2014_ - - - -## Introduction - -I took programming in high school, but I never took to it. This, I strongly believe, is because it wasn't taught right—and teaching it right means starting at the beginning, with unix. The reason for this is three-fold: _(1)_ it gives you a deeper sense of how a high-level computer works (which a glossy front, like Windows, conceals); _(2)_ it's the most natural port of entry into all other programming languages; and _(3)_ it's super-useful in its own right. If you don't know unix and start programming, some things will forever remain hazy and mysterious, even if you can't put your finger on exactly what they are. If you already know a lot about computers, the point is moot; if you don't, then by all means start your programming education by learning unix! - -A word about terminology here: I'm in the habit of horrendously confusing and misusing all of the precisely defined words "[Unix][1]", "[Linux][2]", "[The Command Line][3]", "[The Terminal][4]", "[Shell Scripting][5]", and "[Bash][6]." Properly speaking, _unix_ is an operating system while _linux_ refers to a closely-related family of unix-based operating systems, which includes commercial and non-commercial distributions [1]. (Unix was not free under its developer, AT&T, which caused the unix-linux schism.) The _command line_, as [Wikipedia][3] says, is: - -> ... a means of interacting with a computer program where the user issues commands to the program in the form of successive lines of text (command lines) ... The interface is usually implemented with a command line shell, which is a program that accepts commands as text input and converts commands to appropriate operating system functions. - -So what I mean when I proselytize for "unix", is simply that you learn how to punch commands in on the command line. The _terminal_ is your portal into this world. Here's what my mine looks like: - -![image][7] - - -There is a suite of commands to become familiar with—[The GNU Core Utilities][8] ([wiki entry][9])—and, in the course of learning them, you learn about computers. Unix is a foundational piece of a programming education. - -In terms of bang for the buck, it's also an excellent investment. You can gain powerful abilities by learning just a little. My coworker was fresh out of his introductory CS course, when he was given a small task by our boss. He wrote a full-fledged program, reading input streams and doing heavy parsing, and then sent an email to the boss that began, _"After 1.5 days of madly absorbing perl syntax, I completed the exercise..."_ He didn't know how to use the command-line at the time, and now [a print-out of that email hangs on his wall][10] as a joke—and as a monument to the power of the terminal. - -You can find ringing endorsements for learning the command line from all corners of the internet. For instance, in the excellent course [Startup Engineering (Stanford/Coursera)][11] Balaji Srinivasan writes: - -> A command line interface (CLI) is a way to control your computer by typing in commands rather than clicking on buttons in a graphical user interface (GUI). Most computer users are only doing basic things like clicking on links, watching movies, and playing video games, and GUIs are fine for such purposes. -> -> But to do industrial strength programming - to analyze large datasets, ship a webapp, or build a software startup - you will need an intimate familiarity with the CLI. Not only can many daily tasks be done more quickly at the command line, many others can only be done at the command line, especially in non-Windows environments. You can understand this from an information transmission perspective: while a standard keyboard has 50+ keys that can be hit very precisely in quick succession, achieving the same speed in a GUI is impossible as it would require rapidly moving a mouse cursor over a profusion of 50 buttons. It is for this reason that expert computer users prefer command-line and keyboard-driven interfaces. - -To provide foreshadowing, here are some things you can do in unix: - -* make or rename 100 folders or files _en masse_ -* find all files of a given extension or any file that was created within the last week -* log onto a computer remotely and access its files with ssh -* copy files to your computer directly over the network (no external hard drive necessary!) with [rsync][12] -* run a [Perl][13] or [Python][14] script -* run one of the many programs that are only available on the command line -* see all processes running on your computer or the space occupied by your folders -* see or change the permissions on a file -* parse a text file in any way imaginable (count lines, swap columns, replace words, etc.) -* soundly encrypt your files or communications with [gpg2][15] -* run your own web server on the [Amazon cloud][16] with [nginx][17] -What do all of these have in common? All are hard to do in the [GUI][18], but easy to do on the command line. It would be remiss not to mention that unix is not for everything. In fact, it's not for a lot of things. Knowing which language to use for what is usually a matter of common sense. However, when you start messing about with computers in any serious capacity, you'll bump into unix very quickly—and that's why it's our starting point. - -Is this the truth, the whole truth, and nothing but the truth, [so help my white ass][19]? I believe it is, but also my trajectory through the world of computing began with unix. So perhaps I instinctively want to push this on other people: do it the way I did it. And, because it occupies a bunch of my neuronal real estate at the moment, I could be considered brainwashed :-) - -* * * - - -[1] Still confused about unix vs linux? [Refer to the full family tree][20] and these more precise definitions from Wikipedia: -**_unix_**: _a family of multitasking, multiuser computer operating systems that derive from the original AT&T Unix, developed in the 1970s at the Bell Labs research center by Ken Thompson, Dennis Ritchie, and others_ -**_linux_**: _a Unix-like and mostly POSIX-compliant computer operating system assembled under the model of free and open-source software development and distribution [whose] defining component ... is the Linux kernel, an operating system kernel first released [in] 1991 by Linus Torvalds_ ↑ - -## 100 Useful Unix Commands - -This article is an introduction to unix. It aims to teach the basic principles and neglects to mention many of the utilities that give unix superpowers. To learn about those, see [100 Useful Unix Commands][21]. - -## Getting Started: Opening the Terminal - -If you have a Mac, navigate to Applications > Utilities and open the application named "Terminal": - -![image][22] - -If you have a PC, _abandon all hope, ye who enter here_! Just kidding—partially. None of the native Windows shells, such as [cmd.exe][23] or [PowerShell][24], are unix-like. Instead, they're marked with hideous deformities that betray their ignoble origin as grandchildren of the MS-DOS command interpreter. If you didn't have a compelling reason until now to quit using PCs, here you are [1]. Typically, my misguided PC friends don't use the command line on their local machines; instead, they have to `ssh` into some remote server running Linux. (You can do this with an ssh client like [PuTTY][25], [Chrome's Terminal Emulator][26], or [MobaXterm][27], but don't ask me how.) On Macintosh you can start practicing on the command line right away without having to install a Linux distribution [2] (the Mac-flavored unix is called [Darwin][28]). - -For both Mac and PC users who want a bona fide Linux command line, one easy way to get it is in the cloud with [Amazon EC2][16] via the [AWS Free Tier][29]. If you want to go whole hog, you can download and install a Linux distribution—[Ubuntu][30], [Mint][31], [Fedora][32], and [CentOs][33] are popular choices—but this is asking a lot of non-hardcore-nerds (less drastically, you could boot Linux off of a USB drive or run it in a virtual box). - -* * * - - -[1] I should admit, you can and should get around this by downloading something like [Cygwin][34], whose homepage states: _"Get that Linux feeling - on Windows"_ ↑ -[2] However, if you're using Mac OS rather than Linux, note that OS does not come with the [GNU coreutils][9], which are the gold standard. [You should download them][35] ↑ - -## The Definitive Guides to Unix, Bash, and the Coreutils - -Before going any further, it's only fair to plug the authoritative guides which, unsurprisingly, can be found right on your command line: - - $ man bash - $ info coreutils - -(The $ at the beginning of the line represents the terminal's prompt.) These are good references, but overwhelming to serve as a starting point. There are also great resources online: although these guides, too, are exponentially more useful once you have a small foundation to build on. - -## The Unix Filestructure - -All the files and _directories_ (a fancy word for "folder") on your computer are stored in a hierarchical tree. Picture a tree in your backyard upside-down, so the trunk is on the top. If you proceed downward, you get to big branches, which then give way to smaller branches, and so on. The trunk contains everything in the sense that everything is connected to it. This is the way it looks on the computer, too, and the trunk is called the _root directory_. In unix it's represented with a slash: - -/ - -The root contains directories, which contain other directories, and so on, just like our tree. To get to any particular file or directory, we need to specify the _path_, which is a slash-delimited address: - -/dir1/dir2/dir3/some_file - -Note that a full path always starts with the root, because the root contains everything. As we'll see below, this won't necessarily be the case if we specify the address in a _relative_ way, with respect to our current location in the filesystem. - -Let's examine the directory structure on our Macintosh. We'll go to the root directory and look down just one level with the unix command tree. (If we tried to look at the whole thing, we'd print out every file and directory on our computer!) We have: - -![image][36] - - -While we're at it, let's also look at the directory /Users/username, which is the specially designated [_home directory][37]_ on the Macintosh: - -![image][38] - - -One thing we notice right away is that the Desktop, which holds such a revered spot in the GUI, is just another directory—simply the first one we see when we turn on our computer. - -If you're on Linux rather than Mac OS, the directory tree might look less like the screenshot above and more like this: - -![image][39] - - -The naming of these folders is not intuitive, but you can read about the role of each one [here][40]. I've arbitrarily traced out the path to /var/log, a location where some programs store their log files. - -If the syntax of a unix path looks familiar, it is. A webpage's [URL][41], with its telltale forward slashes, looks like a unix path with a domain prepended to it. This is not a coincidence! For a simple static website, its structure on the web is determined by its underlying directory structure on the server, so navigating to: - -http://www.example.com/abc/xyz - -will serve you content in the folder _websitepath/abc/xyz_ on the host's computer (i.e., the one owned by _example.com_). Modern dynamic websites are more sophisticated than this, but it's neat to reflect that the whole word has learned this unix syntax without knowing it. - -To learn more, see the [O'Reilly discussion of the unix file structure][42]. - -## The Great Trailing Slash Debate - -Sometimes you'll see directories written with a trailing slash, as in: - -dir1/ - -This helpfully reminds you that the entity is a directory rather than a file, but on the command line using the more compact _dir1_ is sufficient. There are a handful of unix commands which behave slightly differently if you leave the trailing slash on, but this sort of extreme pedantry isn't worth worrying about. - -## Where Are You? - Your _Path_ and How to Navigate through the Filesystem - -When you open up the terminal to browse through your filesystem, run a program, or do anything, _you're always somewhere_. Where? You start out in the designated _home directory_ when you open up the terminal. The home directory's path is preset by a global variable called HOME. Again, it's /Users/username on a Mac. - -As we navigate through the filesystem, there are some conventions. The _current working directory (cwd)_—whatever directory we happen to be in at the moment—is specified by a dot: - -. - -Sometimes it's convenient to write this as: - -./ - -which is not to be confused with the root directory: - -/ - -When a program is run in the cwd, you often see the syntax: - - $ ./myprogram - -which emphasizes that you're executing a program from the current directory. The directory one above the cwd is specified by two dots: - -.. - -With the trailing slash syntax, that's: - -../ - -A tilde is shorthand for the home directory: - -~ - -or: - -~/ - -To see where we are, we can _print working directory_: - - $ pwd - -To move around, we can _change directory_: - - $ cd /some/path - -By convention, if we leave off the argument and just type: - - $ cd - -we will go home. To _make directory_—i.e., create a new folder—we use: - - $ mkdir - -As an example, suppose we're in our home directory, /Users/username, and want to get one back to /Users. We can do this two ways: - - $ cd /Users - -or: - - $ cd .. - -This illustrates the difference between an _absolute path_ and a _relative path_. In the former case, we specify the complete address, while in the later we give the address with respect to our _cwd_. We could even accomplish this with: - - $ cd /Users/username/.. - -or maniacally seesawing back and forth: - - $ cd /Users/username/../username/.. - -if our primary goal were obfuscation. This distinction between the two ways to specify a path may seem pedantic, but it's not. Many scripting errors are caused by programs expecting an absolute path and receiving a relative one instead or vice versa. Use relative paths if you can because they're more portable: if the whole directory structure gets moved, they'll still work. - -Let's mess around. We know cd with no arguments takes us home, so try the following experiment: - - $ echo $HOME # print the variable HOME - /Users/username - $ cd # cd is equivalent to cd $HOME - $ pwd # print working directory shows us where we are - /Users/username - - $ unset HOME # unset HOME erases its value - $ echo $HOME - - $ cd /some/path # cd into /some/path - $ cd # take us HOME? - $ pwd - /some/path - -What happened? We stayed in /some/path rather than returning to /Users/username. The point? There's nothing magical about _home_—it's merely set by the variable HOME. More about variables soon! - -## Gently Wading In - The Top 10 Indispensable Unix Commands - -Now that we've dipped one toe into the water, let's make a list of the 10 most important unix commands in the universe: - -1. pwd -2. ls -3. cd -4. mkdir -5. echo -6. cat -7. cp -8. mv -9. rm -10. man -Every command has a help or _manual_ page, which can be summoned by typing man. To see more information about pwd, for example, we enter: - - $ man pwd - -But pwd isn't particularly interesting and its man page is barely worth reading. A better example is afforded by one of the most fundamental commands of all, ls, which lists the contents of the _cwd_ or of whatever directories we give it as arguments: - - $ man ls - -The man pages tend to give TMI (too much information) but the most important point is that commands have _flags_ which usually come in a _one-dash-one-letter_ or _two-dashes-one-word_ flavor: - - command -f - command --flag - -and the docs will tell us what each option does. You can even try: - - $ man man - -Below we'll discuss the commands in the top 10 list in more depth. - -## ls - -Let's go HOME and try out [ls][43] with various flags: - - $ cd - $ ls - $ ls -1 - $ ls -hl - $ ls -al - -Some screen shots: - -![image][44] - - - -![image][45] - - -First, vanilla ls. We see our files—no surprises. And ls -1 merely displays our files in a column. To show the human-readable, long form we stack the -h and -l flags: - -ls -hl - -This is equivalent to: - -ls -h -l - -Screenshot: - -![image][46] - - -This lists the owner of the file; the group to which he belongs (_staff_); the date the file was created; and the file size in human-readable form, which means bytes will be rounded to kilobytes, gigabytes, etc. The column on the left shows _permissions_. If you'll indulge mild hyperbole, this simple command is already revealing secrets that are well-hidden by the GUI and known only to unix users. In unix there are three spheres of permission—_user_, _group_, and _other/world_—as well as three particular types for each sphere—_read_, _write_, and _execute_. Everyone with an account on the computer is a unique _user_ and, although you may not realize it, can be part of various groups, such as a particular lab within a university or team in a company. To see yourself and what groups you belong to, try: - - $ whoami - $ groups - -(To see more information about a user, [finger][47] his username.) A string of dashes displays permission: - - --------- - rwx------ - rwxrwx--- - rwxrwxrwx - -This means, respectively: no permission for anybody; read, write, execute permission for only the user; _rwx_ permission for the user and anyone in the group; and _rwx_ permission for the user, group, and everybody else. Permission is especially important in a shared computing environment. You should internalize now that two of the most common errors in computing stem from the two _P_ words we've already learned: _paths and permissions_. The command chmod, which we'll learn later, governs permission. - -If you look at the screenshot above, you see a tenth letter prepended to the permission string, e.g.: - -This has nothing to do with permissions and instead tells you about the type of entity in the directory: _d_ stands for directory, _l_ stands for symbolic link, and a plain dash denotes a file. - -The -a option in: - -ls -al - -lists _all_ files in the directory, including [_dotfiles][48]_. These are files that begin with a dot and are hidden in the GUI. They're often system files—more about them later. Screenshot: - -![image][49] - - -Note that, in contrast to ls -hl, the file sizes are in pure bytes, which makes them a little hard to read. - -A general point about unix commands: they're often robust. For example, with ls you can use an arbitrary number of arguments and it obeys the convention that an asterisk matches anything (this is known as [file _globbing_][50], and I think of it as the prequel to _regular expressions_). Take: - - $ ls . dir1 .. dir2/*.txt dir3/A*.html - -This monstrosity would list anything in the _cwd_; anything in directory _dir1_; anything in the directory one above us; anything in directory _dir2_ that ends with _.txt_; and anything in directory _dir3_ that starts with _A_ and ends with _.html_. You get the point. - -## Single Line Comments in Unix - -Anything prefaced with a # —that's _pound-space_—is a comment and will not be executed: - - $ # This is a comment. - $ # If we put the pound sign in front of a command, it won't do anything: - $ # ls -hl - -Suppose you write a line of code on the command line and decided you don't want to execute it. You have two choices. The first is pressing _Cntrl-c_, which serves as an "abort mission." The second is jumping to the beginning of the line (_Cntrl-a_) and adding the pound character. This has an advantage over the first method that the line will be saved in bash history (discussed below) and can thus be retrieved and modified later. - -In a script, pound-special-character (like _#!_) is sometimes interpreted (see below), so take note and include a space after # to be safe. - -## The Primacy of Text Files, Text Editors - -As we get deeper into unix, we'll frequently be using text editors to edit code, and viewing either data or code in text files. When I got my hands on a computer as a child, I remember text editors seemed like the most boring programs in the world (compared to, say, 1992 [Prince of Persia][51]). And text files were on the bottom of my food chain. But the years have changed me and now I like nothing better than a clean, unformatted _.txt_ file. It's all you need! If you store your data, your code, your correspondence, your book, or almost anything in _.txt_ files with a systematic structure, they can be parsed on the command line to reveal information from many facets. Here's some advice: do all of your text-related work in a good text editor. Open up clunky [Microsoft Word][52], and you've unwittingly spoken a demonic incantation and summoned the beast. Are these the words of a lone lunatic dispensing [hateration][53]? No, because on the command line you can count the words in a text file, search it with [grep][54], input it into a Python program, et cetera. However, a file in Microsoft Word's proprietary and unknown formatting is utterly unusable. - -Because text editors are extremely important, some people develop deep relationships with them. My co-worker, who is a [Vim][55] aficionado, turned to me not long ago and said, "You know how you should think about editing in Vim? _As if you're talking to it._" On the terminal, a ubiquitous and simple editor is [nano][56]. If you're more advanced, try [Vim][55] or [Emacs][57]. Not immune to my co-worker's proselytizing, I've converted to Vim. Although it's sprawling and the learning curve can be harsh—Vim is like a programming language in itself—you can do a zillion things with it. I put together a quick and dirty Vim wiki [here][58]. - -On the GUI, there are many choices: [Sublime][59], [Aquamacs][60], [Smultron][61], etc. I used to use Smultron until I found, unforgivably, that the spacing of documents when you looked at them in the editor and on the terminal was different. I hear good things about Sublime and Aquamacs. - -_Exercise_: Let's try making a text file with nano. Type: - - $ nano file.txt - -and make the following three-row two-column file: (It's _Cntrl-o_ to save and _Cntrl-x_ to exit.) - -## _echo_ and _cat_ - -More essential commands: echo prints the _string_ passed to it as an argument, while cat prints the _contents_ of files passed to it as arguments. For example: - - $ echo joe - $ echo "joe" - -would both print _joe_, while: - - $ cat file.txt - -would print the contents of _file.txt_. Entering: - - $ cat file.txt file2.txt - -would print out the contents of both _file.txt_ and _file2.txt_ concatenated together, which is where this command gets its slightly confusing name. - -Finally, a couple of nice flags for these commands: - - $ echo -n "joe" # suppress newline - $ echo -e "joetjoenjoe" # interpret special chars ( t is tab, n newline ) - $ cat -n file.txt # print file with line numbers - -## _cp_, _mv_, and _rm_ - -Finishing off our top 10 list we have cp, mv, and rm. The command to make a copy of a file is cp: - - $ cp file1 file2 - $ cp -R dir1 dir2 - -The first line would make an identical copy of _file1_ named _file2_, while the second would do the same thing for directories. Notice that for directories we use the -R flag (for _recursive_). The directory and everything inside it are copied. - -_Question_: what would the following do? - - $ cp -R dir1 ../../ - -_Answer_: it would make a copy of _dir1_ up two levels from our current working directory. - -To rename a file or directory we use mv: - - $ mv file1 file2 - -In a sense, this command also moves files, because we can rename a file into a different path. For example: - - $ mv file1 dir1/dir2/file2 - -would move _file1_ into _dir1/dir2/_ and change its name to _file2_, while: - - $ mv file1 dir1/dir2/ - -would simply move _file1_ into _dir1/dir2/_ or, if you like, rename _./file1_ as _./dir1/dir2/file1_. - -Finally, rm removes a file or directory: - - $ rm file # removes a file - $ rm -r dir # removes a file or directory - $ rm -rf dir # force removal of a file or directory - # (i.e., ignore warnings) - -## Variables in Unix - -To declare something as a variable use an equals sign, with no spaces. Let's declare _a_ to be a variable: - - $ a=3 # This syntax is right (no whitespace) - $ a = 3 # This syntax is wrong (whitespace) - -bash: a: command not found - -Once we've declared something as a variable, we need to use _$_ to access its value (and to let bash know it's a variable). For example: - - $ a=3 - $ echo a - a - $ echo $a - 3 - -So, with no _$_ sign, bash thinks we just want to echo the string _a_. With a _$_ sign, however, it knows we want to access what the variable _a_ is storing, which is the value _3_. Variables in unix are loosely-typed, meaning you don't have to declare something as a string or an integer. - - $ a=3 # a can be an integer - $ echo $a - 3 - - $ a=joe # or a can be a string - $ echo $a - joe - - $ a="joe joe" # Use quotes if you want a string with spaces - $ echo $a - joe joe - -We can declare and echo two variables at the same time, and generally play fast and loose, as we're used to doing on the command line: - - $ a=3; b=4 - $ echo $a $b - 3 4 - $ echo $a$b # mesh variables together as you like - 34 - $ echo "$a$b" # use quotes if you like - 34 - $ echo -e "$at$b" # the -e flag tells echo to interpret t as a tab - 3 4 - -You should also be aware of how bash treats double vs single quotes. As we've seen, if you want to use a string with spaces, you use double quotes. If you use double quotes, any variable inside them will be expanded, the same as in Perl. If you use single quotes, everything is taken literally and variables are not expanded. Here's an example: - - $ var=5 - $ joe=hello $var - -bash: 5: command not found - - $ joe="hello $var" - $ echo $joe - hello 5 - - $ joe='hello $var' - $ echo $joe - hello $var - -An important note is that often we use variables to store _paths_ in unix. Once we do this, we can use all of our familiar directory commands on the variable: - - $ d=dir1/dir2/dir3 - $ ls $d - $ cd $d - - $ d=.. # this variable stores the directory one above us (relative path) - $ cd $d/.. # cd two directories up - -## Escape Sequences - -[Escape sequences][62] are important in every language. When bash reads _$a_ it interprets it as whatever's stored in the variable _a_. What if we actually want to echo the string _$a_? To do this, we use as an escape character: - - $ a=3 - $ echo $a - 3 - $ echo $a - $a - $ echo "$a" # use quotes if you like - $a - -What if we want to echo the slash, too? Then we have to escape the escape character (using the escape character!): - - $ echo \$a # escape the slash and the dollar sign - $a - -This really comes down to parsing. The slash helps bash figure out if your text is a plain old string or a variable. It goes without saying that you should avoid special characters in your variable names. In unix we might occasionally fall into a parsing tar-pit trap. To avoid this, and make extra sure bash parses our variable right, we can use the syntax _${a}_ as in: - - $ echo ${a} - 3 - -When could this possibly be an issue? Later, when we discuss scripting, we'll learn that _$n_, where _n_ is a number, is the _n_th argument to our script. If you were crazy enough to write a script with 11 arguments, you'd discover that bash interprets a=$11 as a=$1 (the first argument) concatenated with the string 1 while a=${11} properly represents the eleventh argument. This is getting in the weeds, but FYI. - -Here's a more practical example: - - $ a=3 - $ echo $a # variable a equals 3 - 3 - $ echo $apple # variable apple is not set - - $ echo ${a}pple # this describes the variable a plus the string "pple" - 3pple - -## Global Variables in Unix - -In general, it is the convention to use capital letters for global variables. We've already learned about one: HOME. We can see all the variables set in our shell by simply typing: - - $ set - -Some basic variables deserve comment: - -* HOME -* PS1 -* TMPDIR -* EDITOR -* DISPLAY -HOME, as we've already seen, is the path to our home directory (preset to /Users/username on Macintosh). PS1 sets the shell's prompt. For example: - - $ PS1=':-) ' - -changes our prompt from a dollar-sign into an emoticon, as in: - -![image][63] - - -On your computer there is a designated temporary directory and its path is stored in TMPDIR. Some commands, such as sort, which we'll learn later, surreptitiously make use of this directory to store intermediate files. At work, we have a shared computer system and occasionally this common directory $TMPDIR will run out of space, causing programs trying to write there to fail. One solution is to simply set TMPDIR to a different path where there's free space. EDITOR sets the default text editor (you can invoke it by pressing _Cntrl-x-e_). And DISPLAY is a variable related to the [X Window System][64]. - -Many programs rely on their own agreed-upon global variables. For example, if you're a Perl user, you may know that Perl looks for modules in the directory whose path is stored in PERL5LIB. Python looks for its modules in PYTHONPATH; R looks for packages in R_LIBS; Matlab uses MATLABPATH; awk uses AWKPATH; C++ looks for libraries in LD_LIBRARY_PATH; and so on. These variables don't exist in the shell by default. A program will make a system call and look for the variable. If the user has had the need or foresight to define it, the program can make use of it. - -## The _PATH_ - -The most important global variable of all is the PATH. This is _the_ PATH, as distinct from _a_ path, a term we've already learned referring to a location in the filesystem. The PATH is a colon-delimited list of directories where unix will look for executable programs when you enter something on command line. If your program is in one of these directories, you can run it from any location by simply entering its name. If the program is not in one of these directories, you can still run it, of course, but you'll have to include its path. - -Let's revisit the idea of a command in unix. What's a command? It's nothing more than a program sitting in a directory somewhere. So, if ls is a program, where is it? Use the command which to see its path: - - $ which ls # on my work computer - /bin/ls - - $ which ls # on my home Mac - /usr/local/Cellar/coreutils/8.20/libexec/gnubin/ls - -For the sake of argument, let's say I download an updated version of the ls command, and then type ls in my terminal. What will happen—will the old ls or the new ls execute? The PATH comes into play here because it also determines priority. When you enter a command, unix will look for it in each directory of the PATH, from first to last, and execute the first instance it finds. For example, if: - -PATH=/bin/dir1:/bin/dir2:/bin/dir3 - -and there's a command named _ls_ in both /bin/dir1 and /bin/dir2, the one in /bin/dir1 will be executed. - -Let's see what your PATH looks like. Enter: - - $ echo $PATH - -For example, here's a screenshot of the default PATH on Ubuntu: - -![image][65] - - -To emphasize the point again, all the programs in the directories specified by your PATH are all the programs that you can access on the command line by simply typing their names. - -The PATH is not immutable. You can set it to be anything you want, but in practice you'll want to augment, rather than overwrite, it. By default, it contains directories where unix expects executables, like: - -* /bin -* /usr/bin -* /usr/local/bin -Let's say you have just written the command /mydir/newcommand. If you're not going to use the command very often, you can invoke it using its full path every time you need it: - - $ /mydir/newcommand - -However, if you're going to be using it frequently, you can just add /mydir to the PATH and then invoke the command by name: - - $ PATH=/mydir:$PATH # add /mydir to the front of PATH - highest priority - $ PATH=$PATH:/mydir # add /mydir to the back of PATH - lowest priority - $ newcommand # now invoking newcommand is this easy - -This is a frequent chore in unix. If you download some new program, you will often find yourself updating the PATH to include the directory containing its binaries. How can we avoid having to do this every time we open the terminal for a new session? We'll discuss this below when we learn about _.bashrc_. - -If you want to shoot yourself in the foot, you can vaporize the PATH: - - $ unset PATH # not advisable - $ ls # now ls is not found - -bash: ls: No such file or directory - -but this is not advisable, save as a one-time educational experience. - -## Links - -While we're on the general subject of paths, let's talk about [_symbolic links][66]_. If you've ever used the _Make Alias_ command on a Macintosh (not to be confused with the unix command alias, discussed below), you've already developed intuition for what a link is. Suppose you have a file in one folder and you want that file to exist in another folder simultaneously. You could copy the file, but that would be wasteful. Moreover, if the file changes, you'll have to re-copy it—a huge ball-ache. Links solve this problem. A link to a file is a stand-in for the original file, often used to access the original file from an alternate file path. It's not a copy of the file but, rather, points to the file. - -To make a symbolic link, use the command [ln][67]: - - $ ln -s /path/to/target/file mylink - -This produces: - - mylink --> /path/to/target/file - -in the cwd, as ls -hl will show. Note that removing _mylink_: - - $ rm mylink - -does not affect our original file. - -If we give the target (or source) path as the sole argument to ln, the name of the link will be the same as the source file's. So: - - $ ln -s /path/to/target/file - -produces: - - file --> /path/to/target/file - -Links are incredibly useful for all sorts of reasons—the primary one being, as we've already remarked, if you want a file to exist in multiple locations without having to make extraneous, space-consuming copies. You can make links to directories as well as files. Suppose you add a directory to your PATH that has a particular version of a program in it. If you install a newer version, you'll need to change the PATH to include the new directory. However, if you add a link to your PATH and keep the link always pointing to the most up-to-date directory, you won't need to keep fiddling with your PATH. The scenario could look like this: - - $ ls -hl myprogram - current -> version3 - version1 - version2 - version3 - -(where I'm hiding some of the output in the long listing format.) In contrast to our other examples, the link is in the same directory as the target. Its purpose is to tell us which version, among the many crowding a directory, we should use. - -Another good practice is putting links in your home directory to folders you often use. This way, navigating to those folders is easy when you log in. If you make the link: - - ~/MYLINK --> /some/long/and/complicated/path/to/an/often/used/directory - -then you need only type: - - $ cd MYLINK - -rather than: - - $ cd /some/long/and/complicated/path/to/an/often/used/directory - -Links are everywhere, so be glad you've made their acquaintance! - -## What is Scripting? - -By this point, you should be comfortable using basic utilities like echo, cat, mkdir, cd, and ls. Let's enter a series of commands, creating a directory with an empty file inside it, for no particular reason: - - $ mkdir tmp - $ cd tmp - $ pwd - /Users/oliver/tmp - $ touch myfile.txt # the command touch creates an empty file - $ ls - myfile.txt - $ ls myfile_2.txt # purposely execute a command we know will fail - ls: cannot access myfile_2.txt: No such file or directory - -What if we want to repeat the exact same sequence of commands 5 minutes later? _Massive bombshell_—we can save all of these commands in a file! And then run them whenever we like! Try this: - - $ nano myscript.sh - -and write the following: - - # a first script - mkdir tmp - cd tmp - pwd - touch myfile.txt - ls - ls myfile_2.txt - -Gratuitous screenshot: - -![image][68] - - -This file is called a _script_ (_.sh_ is a typical suffix for a shell script), and writing it constitutes our first step into the land of bona fide computer programming. In general usage, a script refers to a small program used to perform a niche task. What we've written is a recipe that says: ** - -* create a directory called "tmp" -* go into that directory -* print our current path in the file system -* make a new file called "myfile.txt" -* list the contents of the directory we're in -* specifically list the file "myfile_2.txt" (which doesn't exist) -** This script, though silly and useless, teaches us the fundamental fact that all computer programs are ultimately just lists of commands. - -Let's run our program! Try: - - $ ./myscript.sh - -bash: ./myscript.sh: Permission denied - -_WTF!_ It's dysfunctional. What's going on here is that the file permissions are not set properly. In unix, when you create a file, the default permission is _not executable_. You can think of this as a brake that's been engaged and must be released before we can go (and do something potentially dangerous). First, let's look at the file permissions: - - $ ls -hl myscript.sh - -rw-r--r-- 1 oliver staff 75 Oct 12 11:43 myscript.sh - -Let's change the permissions with the command chmod and execute the script: - - $ chmod u+x myscript.sh # add executable(x) permission for the user(u) only - $ ls -hl myscript.sh - -rwxr--r-- 1 oliver staff 75 Oct 12 11:43 myscript.sh - - $ ./myscript.sh - /Users/oliver/tmp/tmp - myfile.txt - ls: cannot access myfile_2.txt: No such file or directory - -Not bad. Did it work? Yes, it did because it's printed stuff out and we see it's created tmp/myfile.txt: - - $ ls - myfile.txt myscript.sh tmp - $ ls tmp - myfile.txt - -An important note is that even though there was a cd in our script, if we type: - - $ pwd - /Users/oliver/tmp - -we see that we're still in the same directory as we were in when we ran the script. Even though the script entered /Users/oliver/tmp/tmp, and did its bidding, we stay in /Users/oliver/tmp. Scripts always work this way—where they go is independent of where we go. - -If you're wondering why anyone would write such a pointless script, you're right—it would be odd if we had occasion to repeat this combination of commands. There are some more realistic examples of scripting below. - -## File Suffixes in Unix - -As we begin to script it's worth following some file naming conventions. We should use common sense suffixes, like: - -* _.txt_ \- for text files -* _.html_ \- for html files -* _.sh_ \- for shell scripts -* _.pl_ \- for Perl scripts -* _.py_ \- for Python scripts -* _.cpp_ \- for c++ code -and so on. Adhering to this organizational practice will enable us to quickly scan our files, and make searching for particular file types easier [1]. As we saw above, commands like ls and find are particularly well-suited to use this kind of information. For example, list all text files in the cwd: - - $ ls *.txt - -List all text files in the cwd and below (i.e., including child directories): - - $ find . -name "*.txt" - - - -* * * - - -[1] An astute reader noted that, for commands—as opposed to, say, html or text files—using suffixes is not the best practice because it violates the principle of encapsulation. The argument is that a user is neither supposed to know nor care about a program's internal implementation details, which the suffix advertises. You can imagine a program that starts out as a shell script called _mycommand.sh_, is upgraded to Python as _mycommand.py_, and then is rewritten in C for speed, becoming the binary _mycommand_. What if other programs depend on _mycommand_? Then each time _mycommand_'s suffix changes they have to be rewritten—a big problem. Although I make this sloppy mistake in this article, that doesn't excuse you! [Read the full argument][69] ↑ - -## The Shebang - -We've left out one important detail about scripting. How does unix know we want to run a bash script, as opposed to, say, a Perl or Python script? There are two ways to do it. We'll illustrate with two simple scripts, a bash script and a Perl script: - - $ cat myscript_1.sh # a bash script - echo "hello kitty" - - $ cat myscript_1.pl # a Perl script - print "hello kittyn"; - -The first way to tell unix which program to use to interpret the script is simply to say so on the command line. For example, we can use bash to execute bash scripts: - - $ bash ./myscript_1.sh # use bash for bash scripts - hello kitty - -and perl for Perl scripts: - - $ perl ./myscript_1.pl # use Perl for Perl scripts - hello kitty - -But this won't work for a Perl script: - - $ ./myscript_1.pl # this won't work - ./myscript_1.pl: line 1: print: command not found - -And if we purposefully specify the wrong language, we'll get errors: - - $ bash ./myscript_1.pl # let's purposefully do it backwards - ./myscript_1.pl: line 1: print: command not found - - $ perl ./myscript_1.sh - String found where operator expected at ./myscript_1.sh line 1, - near "echo "hello kitty"" - (Do you need to predeclare echo?) - syntax error at ./myscript_1.sh line 1, near "echo "hello kitty"" - Execution of ./myscript_1.sh aborted due to compilation errors. - -The second way to specify the proper interpreter—and the better way, which you should emulate—is to put it in the script itself using a [_shebang_][70]. To do this, let's remind ourselves where bash and perl reside on our system. On my computer, they're here: - - $ which perl - /usr/bin/perl - - $ which bash - /bin/bash - -although perl could be somewhere else on your machine (bash should be in /bin by convention). The _shebang_ specifies the language in which your script is interpreted according to the syntax #! followed by the path to the language. It should be the first line of your script. Note that it's not a comment even though it looks like one. Let's add shebangs to our two scripts: - - $ cat myscript_1.sh - #!/bin/bash - echo "hello kitty" - - $ cat myscript_1.pl - #!/usr/bin/perl - print "hello kittyn"; - -Now we can run them without specifying the interpreter in front: - - $ ./myscript_1.sh - hello kitty - $ ./myscript_1.pl - hello kitty - -However, there's _still_ a lingering issue and it has to do with [portability][71], an important software principle. What if perl is in a different place on your machine than mine and you copy my scripts and try to run them? The path will be wrong and they won't work. The solution to this issue is courtesy of a neat trick using [env][72]. We can amend our script to be: - - $ cat myscript_1.pl - #!/usr/bin/env perl - print "hello kittyn"; - -Of course, this assumes you have a copy of env in /usr/bin, but this is usually a correct assumption. What env does here is to use whatever your environmental variable for perl is—i.e., the perl that's first in your PATH. - -This is a useful practice even if you're not sharing scripts. Suppose you've updated your version of perl and there's a newer copy than /usr/bin/perl. You've appropriately updated your PATH such that the directory containing the updated perl comes before /usr/bin. If you have env in your shebang, you're all set. However, if you've _hardwired_ the old path in your shebang, your script will run on the old perl [1]. - -The question that the shebang resolves—which program will run your script?—reminds us of a more fundamental distinction between [interpreted languages][73] and [compiled languages][74]. The former are those like bash, Perl, and Python, where you can cat a script and look inside it. The later, like C++, require [_compilation][75]_, the process whereby code is translated into machine language (the result is sometimes called a _binary_). This can be done with a command line utility like [g++][76]: - - $ g++ MyProgram.cpp -o MyProgram - -Compiled programs, such as the unix utilities themselves, tend to run faster. Don't try to cat a binary, such as ls, or it will spew out gibberish: - - $ cat $( which ls ) # don't do this! - - - -* * * - - -[1] Of course, depending on circumstances, you may very well want to stick with the old version of Perl or whatever's running your program. An update can have unforeseen consequences and this is the motivation for tools like [virtualenv][77] (Python), whose docs remind us: "_If an application works, any change in its libraries or the versions of those libraries can break the application_" ↑ - -## _bash_ - -We've thrown around the term _bash_ a few times but we haven't defined it. To do so, let's examine the special command, sh, which is more primitive than bash and came before it. To quote Wikipedia and the manual page: - -> The Bourne shell (sh) is a shell, or command-line interpreter, for computer operating systems. The shell is a command that reads lines from either a file or the terminal, interprets them, and generally executes other commands. It is the program that is running when a user logs into the system ... Commands can be typed directly to the running shell or can be put into a file and the file can be executed directly by the shell - -As it describes, sh is special because it's both a command interpreter and a command itself (usually found at /bin/sh). Put differently, you can run _myscript_ as: - - $ sh ./myscript - -or you can simply type: - - $ sh - -to start an interactive sh shell. If you're in this shell and run: - - $ ./myscript - -without specifying an interpreter or using a shebang, your script will be interpreted by sh by default. On most computers, however, the default shell is no longer sh but bash (usually located at /bin/bash). To mash up Wikipedia and the manual page: - -> The **B**ourne-**A**gain **SH**ell (bash) a Unix shell written by Brian Fox for the GNU Project as a free software replacement for the Bourne shell. bash is an sh-compatible command language interpreter that executes commands read from the standard input or from a file ... There are some subtle differences between bash and traditional versions of sh - -Like sh, bash is a command you can either invoke on a script or use to start an interactive bash shell. Read more on Stackoverflow: [Difference between sh and bash][78]. - -Which shell are you using right now? Almost certainly bash, but if you want to double check, there's a neat command [given here][79] to display your shell type: - - $ ps -p $$ - -There are more exotic shells, like [Z shell][80] and [tcsh][81], but they're beyond the scope of this article. - -## _chmod_ - -Let's take a closer look at how to use [chmod][82]. Remember the three domains: - -* _u_ \- user -* _g_ \- group -* _o_ \- other/world -and the three types of permission: -* _r_ \- read -* _w_ \- write -* _x_ \- execute -we can mix and match these how we like, using a plus sign to grant permissions according to the syntax: - -chmod entity+permissiontype - -or a minus sign to remove permissions: - -chmod entity-permissiontype - -E.g.: - - $ chmod u+x myfile # make executable for you - $ chmod g+rxw myfile # add read write execute permissions for the group - $ chmod go-wx myfile # remove write execute permissions for the group - # and for everyone else (excluding you, the user) - -You can also use _a_ for "all of the above", as in: - - $ chmod a-rwx myfile # remove all permissions for you, the group, - # and the rest of the world - -If you find the above syntax cumbersome, there's a numerical shorthand you can use with chmod. The only two I have memorized are _777_ and _755_: - - $ chmod 777 myfile # grant all permissions (rwxrwxrwx) - $ chmod 755 myfile # reserve write access for the user, - # but grant all other permissions (rwxr-xr-x) - -Read more about the numeric code [here][83]. In general, it's a good practice to allow your files to be writable by you alone, unless you have a compelling reason to share access to them. - -## _ssh_ - -In addition to chmod, there's another command it would be remiss not to mention. For many people, the first time they need to go to the command line, rather than the GUI, is to use the [Secure Shell (ssh)][84] protocol. Suppose you want to use a computer, but it's not the computer that's in front of you. It's a different computer in some other location—say, at your university, your company, or on the [Amazon cloud][16]. [ssh][85] is the command that allows you to log into a computer remotely over the network. Once you've sshed into a computer, you're in its shell and can run commands on it just as if it were your personal laptop. To ssh, you need to know your user name, the address of the host computer you want to log into, and the password [1]. The basic syntax is: - -ssh username@host - -For example: - - $ ssh username@myhost.university.edu - -If you're trying to ssh into a private computer and don't know the hostname, use its IP address (_username@IP-address_). - -ssh also allows you to run a command on the remote server without logging in. For instance, to list of the contents of your remote computer's home directory, you could run: - - $ ssh username@myhost.university.edu "ls -hl" - -Cool, eh? Moreover, if you have ssh access to a machine, you can copy files to or from it with the utility [rsync][12]—a great way to move data without an external hard drive. - -The file: - -~/.ssh/config - -determines ssh's behavior and you can create it if it doesn't exist (the dot in the name _.ssh_ confers invisibility—[see the discussion about dotfiles below][86]). On your own private computer, you can ssh into selected servers without having to type in a password by updating this configuration file. To do this, generate [rsa][87] ssh [keys][88]: - - $ mkdir -p ~/.ssh - $ cd ~/.ssh - $ ssh-keygen -t rsa -f localkey - -This will create two files on your computer, a public key: - -~/.ssh/localkey.pub - -and a private key: - -~/.ssh/localkey - -You can share your public key, but _do not give anyone your private key!_ Suppose you want to ssh into _myserver.com_. Normally, that's: - - $ ssh myusername@myserver.com - -Instead of doing this, add these lines to your _~/.ssh/config_ file: - - Host Myserver - HostName myserver.com - User myusername - IdentityFile ~/.ssh/localkey - -Next, cat your public key and paste it into: - -~/.ssh/authorized_keys - -on the remote machine (i.e., the _myserver.com_ computer). Now on your local computer, you can ssh into _myserver.com_ without a password: - - $ ssh Myserver - -You can also use this technique to push to [github.com][89] [2], without having to punch your password in each time, by pasting your public key into: - -Settings > SSH Keys > Add SSH Key - -on GitHub (read the [official tutorial][90]). - -If this is your first encounter with ssh, you'd be surprised how much of the work of the world is done by ssh. It's worth reading the extensive man page, which gets into matters of computer security and cryptography. - -* * * - - -[1] The host also has to enable ssh access. On Macintosh, for example, it's disabled by default, but you can turn it on, as instructed [here][91] ↑ -[2] As you get deeper into the game, tracking your scripts and keeping a single, stable version of them becomes crucial. [Git][92], a vast subject for [another tutorial][93], is the neat solution to this problem and the industry standard for version control. On the web [GitHub][94] provides free hosting of script repositories and connects to the command line via the git interface ↑ - -## Saving to a File; Stdout and Stderr - -To save to a file in unix, use an angle bracket: - - $ echo joe > junk.txt # save to file - $ cat junk.txt - joe - -To append to the end of a pre-existing file, use a double angle bracket: - - $ echo joe >> junk.txt # append to already-existing file - $ cat junk.txt - joe - joe - -Returning to our first script, [_myscript.sh][95]_, let's save the output to a file: - - $ ./myscript.sh > out.txt - mkdir: cannot create directory 'tmp': File exists - ls: cannot access myfile_2.txt: No such file or directory - - $ cat out.txt - /Users/oliver/tmp/tmp - myfile.txt - -This is interesting: _out.txt_ has its output. However, not everything went into _out.txt_, because some error messages were echoed to the console. What's going on here is that there are actually two [output streams][96]: _stdout_ (standard out) and _stderr_ (standard error). Look at the following figure from Wikipedia: - -![image][97] - -(Image credit: [Wikipedia: Standard streams][96]) - -Proper output goes into stdout while errors go into stderr. The syntax for saving stderr in unix is 2> as in: - - $ # save the output into out.txt and the error into err.txt - $ ./myscript.sh > out.txt 2> err.txt - $ cat out.txt - /Users/oliver/tmp/tmp - myfile.txt - $ cat err.txt - mkdir: cannot create directory 'tmp': File exists - ls: cannot access myfile_2.txt: No such file or directory - -When you think about it, the fact that output and error are separated is supremely useful. At work, sometimes we parallelize heavily and run 1000 instances of a script. For each instance, the error and output are saved separately. The 758th job, for example, might look like this: - -./myjob --instance 758 > out758.o 2> out758.e - -(I'm in the habit of using the suffixes _.o_ for output and _.e_ for error.) With this technique we can quickly scan through all 1000 _.e_ files and check if their size is 0. If it is, we know there was no error; if not, we can re-run the failed jobs. Some programs are in the habit of echoing run statistics or other information to stderr. This is an unfortunate practice because it muddies the water and, as in the example above, would make it hard to tell if there was an actual error. - -Output vs error is a distinction that many programming languages make. For example, in C++ writing to stdout and stderr is like this: - - cout << "some output" << endl; - cerr << "some error" << endl; - -In Perl it's: - - print STDOUT "some outputn"; - print STDERR "some errorn"; - -In Python it's: - - import sys - sys.stdout.write("some outputn") - sys.stderr.write("some errorn") - -and so on. - -## More on Stdout and Stderr; Redirection - -For the sake of completeness, we should note that you can redirect standard error to standard output and vice versa. Let's make sure we get the syntax of all things pertaining to stdout and stderr right: - - 1> # save stdout to (plain old > also works) - 2> # save stderr to - -as in: - - $ ./myscript.sh 1> out.o 2> out.e - $ ./myscript.sh > out.o 2> out.e # these two lines are identical - -What if we want to choose where things will be printed from _within_ our script? Then we can use the following syntax: - - &1 # standard out stream - &2 # standard error stream - -Let's examine five possible versions of our _Hello Kitty_ script: - - #!/bin/bash - # version 1 - echo "hello kitty" - - #!/bin/bash - # version 2 - echo "hello kitty" > somefile.txt - - #!/bin/bash - # version 3 - echo "hello kitty" > &1 - - #!/bin/bash - # version 4 - echo "hello kitty" > &2 - - #!/bin/bash - # version 5 - echo "hello kitty" > 1 - -Here's how they work: - -* _version 1_ \- echo "hello kitty" to stdout -* _version 2_ \- echo "hello kitty" to the file somefile.txt -* _version 3_ \- same as version 1 -* _version 4_ \- echo "hello kitty" to sterr -* _version 5_ \- echo "hello kitty" to _the file named 1_ -This illustrates the point of the ampersand syntax: it distinguishes between the output streams and files named _1_ or _2_. Let's try running script version 4 as a sanity check to make sure these scripts are working as expected: - - $ # output saved to file but error printed to console - $ ./hellokitty.sh > junk.txt - hello kitty - -_hello kitty_ is indeed stderr because it's echoed to the console, not saved into _junk.txt_. - -This syntax makes it easy to see how we could, e.g., redirect the standard error to standard output: - - $ ./somescript.sh 2> &1 # redirect stderr to stdout - -I rarely have occasion to do this and, although it's not something you need in your introductory unix toolkit, it's good to know. - -## Conditional Logic - -Conditional Logic is a universal feature of programming languages. The basic idea is, _if_ this condition, _then_ do something. It can be made more complex: _if_ this condition, _then_ do something; _else if_ that condition, _then_ do another thing; _else_ (if any other condition), _then_ do yet another thing. Let's see how to implement this in bash: - - $ a=joe - $ if [ $a == "joe" ]; then echo hello; fi - hello - -or: - - $ a=joe - $ if [ $a == "joe" ]; then echo hello; echo hello; echo hello; fi - hello - hello - hello - -The structure is: - -if [ _condition_ ]; then ... ; fi - -Everything between the words then and fi (_if_ backwards in case you didn't notice) will execute if the condition is satisfied. In other languages, this block is often defined by curly brackets: _{ }_. For example, in a Perl script, the same code would be: - - #!/usr/bin/env perl - - my $a="joe"; - - if ( $a eq "joe" ) - { - print "hellon"; - print "hellon"; - print "hellon"; - } - -In bash, _if_ is if, _else_ is else, and _else if_ is elif. In a script it would look like this: - - #!/bin/bash - - a=joe - - if [ $a == "joe" ]; then - echo hello; - elif [ $a == "doe" ]; then - echo goodbye; - else - echo "ni hao"; - fi - -You can also use a case statement to implement conditional logic. See an example of that [here][98]. - -Although I said in the intro that unix is the best place to start your computer science education, I have to admit that the syntax for _if-then_ logic is somewhat unwieldy—even unfriendly. Bash is a bad teaching language for conditional logic, [arrays][99], [hashes][100], etc. But that's only because its element is not heavy-duty programming with lots of functions, numerical operations, sophisticated data structures, and logic. Its mastery is over the quick and dirty, manipulating files and directories, and doing system stuff. I still maintain it's the proper starting point because of its wonderful tools, and because knowing its fundamentals is a great asset. Every language has its place in the programming ecosystem. Back in College, I stumbled on a physics book called [_The Tiger and the Shark: Empirical Roots of Wave-Particle Dualism_][101] by Bruce Wheaton. The book had a great epigraph: - -> It is like a struggle between a tiger and a shark, -each is supreme in his own element, -but helpless in that of the other. -_J.J. Thomson, 1925_ - -In our context, this would read: bash is supreme on the command line, but not inside of a script. - -## File Test Operators; Return or Exit Status - -_File Test Operators_ and _exit status_ are two completely different topics, but since they both go well with if statements, I'll discuss them here. File Test Operators are things you can stick in an if statement to give you information about a file. Two common problems are _(1)_ checking if your file exists and _(2)_ checking if it's non-zero size: Let's create two files, one empty and one not: - - $ touch emptyfile # create an empty file - $ echo joe > nonemptyfile # create a non-empty file - -The operator _-e_ tests for existence and _-s_ tests for non-zero-ness: - - $ file=emptyfile - $ if [ -e $file ]; then echo "exists"; if [ -s $file ]; then echo "non-0"; fi; fi - exists - - $ file=nonemptyfile - $ if [ -e $file ]; then echo "exists"; if [ -s $file ]; then echo "non-0"; fi; fi - exists - non-0 - -Read The Linux Documentation Project's discussion of file test operators [here][102]. - -Changing the subject altogether, you may be familiar with the idea of a return value in computer science. Functions can return a value upon completion. In unix, commands also have a return value or _exit code_, queryable with: - -$? - -This is usually employed to tell the user whether or not the command successfully executed. By convention, successful execution returns 0. For example: - - $ echo joe - joe - $ echo $? # query exit code of previous command - 0 - -Let's see how the exit code can be useful. We'll make a script, _test_exitcode.sh_, such that: - - $ cat test_exitcode.sh - #!/bin/bash - sleep 10 - -This script just pauses for 10 seconds. First, we'll let it run and then we'll interrupt it using _Cntrl-c_: - - $ ./test_exitcode.sh; # let it run - $ echo $? - 0 - - $ ./test_exitcode.sh; # interrupt it - ^C - $ echo $? - 130 - -The non-zero exit code tells us that it's failed. Now we'll try the same thing with an if statement: - - $ ./test_exitcode.sh - $ if [ $? == 0 ]; then echo "program succeeded"; else echo "program failed"; fi - program succeeded - - $ ./test_exitcode.sh; - ^C - $ if [ $? == 0 ]; then echo "program succeeded"; else echo "program failed"; fi - program failed - -In research, you might run hundreds of command-line programs in parallel. For each instance, there are two key questions: _(1)_ Did it finish? _(2)_ Did it run without error? Checking the exit status is the way to address the second point. You should always check the program you're running to find information about its exit code, since some use different conventions. Read The Linux Documentation Project's discussion of exit status [here][103]. - -_Question_: What's going on here? - - $ if echo joe; then echo joe; fi - joe - joe - -This is yet another example of bash allowing you to stretch syntax like silly putty. In this code snippet, - -echo joe - -is run, and its successful execution passes a _true_ return code to the if statement. So, the two _joe_s we see echoed to the console are from the statement to be evaluated and the statement inside the conditional. We can also invert this formula, doing something if our command fails: - - $ outputdir=nonexistentdir # set output dir equal to a nonexistent dir - $ if ! cd $outputdir; then echo "couldnt cd into output dir"; fi - -bash: pushd: nonexistentdir: No such file or directory - couldnt cd into output dir - - $ mkdir existentdir # make a test directory - $ outputdir=existentdir - $ if ! cd $outputdir; then echo "couldnt cd into output dir"; fi - $ # no error - now we're in the directory existentdir - -Did you follow that? (! means logical NOT in unix.) The idea is, we try to cd but, if it's unsuccessful, we echo an error message. This is a particularly useful line to include in a script. If the user gives an output directory as an argument and the directory doesn't exist, we exit. If it does exist, we cd into it and it's business as usual: - -if ! cd $outputdir; then echo "[error] couldn't cd into output dir"; exit; fi - -Without this line, the script will run in whatever directory it's in if cd fails. Once in lab, I was running a script that didn't have this kind of protection. The output directory wasn't found and the script starting making and deleting files in the wrong directory. It was powerfully uncool! - -We can implement similar constructions using the && and || operators rather than an if statement. Let's see how this works by making some test files: - - $ touch file{1..4} - $ ls - file1 file2 file3 file4 - -The && operator will chug through a chain of commands and keep on going until one of the commands fails, as in: - - $ ( ls file1 ) && ( ls file2 ) && ( ls file3 ) && ( ls file4 ) - file1 - file2 - file3 - file4 - - $ ( ls file1 ) && ( ls file2 ) && ( ls fileX ) && ( ls file4 ) - file1 - file2 - ls: cannot access fileX: No such file or directory - -In contrast, the || operator will proceed through the command chain and _stop_ after the first successful one, as in: - - $ ( ls file1 ) || ( ls file2 ) || ( ls file3 ) || ( ls file4 ) - file1 - - $ ( ls fileX ) || ( ls fileY ) || ( ls fileZ ) || ( ls file4 ) - ls: cannot access fileX: No such file or directory - ls: cannot access fileY: No such file or directory - ls: cannot access fileZ: No such file or directory - file4 - -## Basic Loops - -In programming, loops are a way of performing operations iteratively. Loops come in different flavors, but the _for loop_ and _while loop_ are the most basic. In bash, we can implement a for loop like this: - - $ for i in 1 2 3; do echo $i; done - 1 - 2 - 3 - -The structure is: - -for _variable_ in _list_; do ... ; done - -Put anything you like in the list: - - $ for i in 1 2 hello; do echo $i; done - 1 - 2 - hello - -Many other languages wouldn't let you get away with combining data types in the iterations of a loop, but this is a recurrent bash theme: it's fast; it's loose; it's malleable. - -To count from 1 to 10, try: - - $ for i in {1..10}; do echo -n "$i "; done; echo - 1 2 3 4 5 6 7 8 9 10 - -But if we can just write: - - $ echo {1..10} - -why do we need a loop here? Loops really come into their own in bash when—no surprise!—we're dealing with files, paths, and commands. For example, to loop through all of the text files in the cwd, use: - - $ for i in *.txt; do echo $i; done - -Although this is nearly the same as: - - $ ls *.txt - -the former construction has the advantage that we can stuff as much code as we like in the block between do and done. Let's make a random directory structure like so: - - $ mkdir -p myfolder{1..3}/{X,Y} - -We can populate it with token files (fodder for our example) via a loop: - - $ j=0; for i in myfolder*/*; do echo "*** "$i" ***"; touch ${i}/a_${j}.txt ${i}/b_${j}.txt; ((j++)); done - -In bash, ((j++)) is a way of incrementing j. We echo $i to get some visual feedback as the loop iterates. Now our directory structure looks like this: - -![image][104] - - -To practice loops, suppose we want to find any file that begins with _b_ in any subfolder and make a symbolic link to it from the cwd: - - $ for i in myfolder*/*/b*; do echo "*** "$i" ***"; ln -s $i; done - -As we learned above, a link is not a copy of a file but, rather, a kind of pointer that allows us to access a file from a path other than the one where it actually resides. Our loop yields the links: - - b_0.txt -> myfolder1/X/b_0.txt - b_1.txt -> myfolder1/Y/b_1.txt - b_2.txt -> myfolder2/X/b_2.txt - b_3.txt -> myfolder2/Y/b_3.txt - b_4.txt -> myfolder3/X/b_4.txt - b_5.txt -> myfolder3/Y/b_5.txt - -allowing us to access the _b_ files from the cwd. - -I can't overstate all the heroic things you can do with loops in bash. Suppose we want to change the extension of any text file that begins with _a_ and resides in an _X_ subfolder from _.txt_ to _.html_: - - $ for i in myfolder*/X/a*.txt; do echo "*** "$i" ***"; j=$( echo $i | sed 's|.txt|.html|' ); echo $j; mv $i $j; echo; done - -But I've jumped the gun! This example features three things we haven't learned yet: command substitution, piping, and sed. You should revisit it after reading those sections, but the idea is that the variable _j_ stores a path that looks like our file's but has the extension replaced. And you see that a knowledge of loops is like a stick of dynamite you can use to blow through large numbers of files. - -Here's another contrived example with these yet-to-be-discussed techniques: - - $ for i in $( echo $PATH | tr ":" " " ); do echo "*** "$i" ***"; ls $i | head; echo; done | less - -Can you guess what this does? It shows the first ten commands in each folder in our PATH—not something you'd likely need to do, but a demonstration of the fluidity of these constructions. - -If we want to run a command or script in parallel, we can do that with loops, too. [gzip][105] is a utility to compress files, thereby saving hard drive space. To compress all text files in the cwd, in parallel, do: - - $ for i in *.txt; do { echo $i; gzip $i & }; done - -But I've gotten ahead of myself again. We'll leave the discussion of this example to the section on processes. - -The structure of a while loop is: - -while _condition_; do ... ; done - -I use while loops much less than for loops, but here's an example: - - $ x=1; while ((x <= 3)); do echo $x; ((x++)); done - 1 - 2 - 3 - -The while loop can also take input from a file. Suppose there's a file _junk.txt_ such that: - - $ cat junk.txt - 1 - 2 - 3 - -You can iterate over this file as such: - - $ while read x; do echo $x; done < junk.txt - 1 - 2 - 3 - -## Arguments to a Script - -Now that we've covered basic [control flow][106], let's return to the subject of scripting. An important question is, how can we pass arguments to our script? Let's make a script called _hellokitty.sh_: - - #!/bin/bash - - echo hello - -Try running it: - - $ chmod 755 hellokitty.sh - $ ./hellokitty.sh - hello - -We can change it to the following: - - #!/bin/bash - - echo hello $1 - -Now: - - $ ./hellokitty.sh kitty - hello kitty - -In bash $1 represents the first argument to the script, $2 the second, and so on. If our script is: - - #!/bin/bash - - echo $0 - echo hello $1 $4 - -Then: - - $ ./hellokitty.sh my sweet kitty cat - ./hellokitty.sh - hello my cat - -In most programming languages, arguments passed in on the command line are stored as an array. Bash stores the _n_th element of this array in the variable $_n_. $0 is special and refers to the name of the script itself. - -For casual scripts this suits us well. However, as you go on to write more involved programs with many options, it becomes impractical to rely on the position of an argument to determine its function in your script. The proper way to do this is using _flags_ that can be deployed in arbitrary order, as in: - -command --flag1 1 --flag2 1 --flag3 5 - -or, in short form: - -command -f1 1 -f2 1 -f3 5 - -You can do this with the command [getopts][107], but it's sometimes easier just to write your own options parser. Here's a sample script called [_test_args][108]_. Although a case statement would be a good way to handle numerous conditions, I'll use an if statement: - - #!/bin/bash - - helpmessage="This script showcases how to read arguments" - - ### get arguments - # while input array size greater than zero - while (($# > 0)); do - if [ "$1" == "-h" -o "$1" == "-help" -o "$1" == "--help" ]; then - shift; - echo "$helpmessage" - exit; - elif [ "$1" == "-f1" -o "$1" == "--flag1" ]; then - # store what's passed via flag1 in var1 - shift; var1=$1; shift - elif [ "$1" == "-f2" -o "$1" == "--flag2" ]; then - shift; var2=$1; shift - elif [ "$1" == "-f3" -o "$1" == "--flag3" ]; then - shift; var3=$1; shift - # if unknown argument, just shift - else - shift - fi - done - - ### main - # echo variable if not empty - if [ ! -z $var1 ]; then echo "flag1 passed "$var1; fi - if [ ! -z $var2 ]; then echo "flag2 passed "$var2; fi - if [ ! -z $var3 ]; then echo "flag3 passed "$var3; fi - -This has some things we haven't seen yet: - -* $# is the size of our input argument array -* shift pops an element off of our array (the same as in Perl) -* exit exits the script -* -o is logical OR in unix -* -z checks if a variable is empty -The code loops through the argument array and keeps popping off elements until the array size is zero, whereupon it exits the loop. For example, one might run this script as: - - $ ./test_args --flag1 x -f2 y --flag3 zzz - flag1 passed x - flag2 passed y - flag3 passed zzz - -To spell out how this works, the first argument is _\--flag1_. Since this matches one of our checks, we shift. This pops this element out of our array, so the first element, $1, becomes _x_. This is stored in the variable _var1_, then there's another shift and $1 becomes _-f2_, which matches another condition, and so on. - -The flags can come in any order: - - $ ./test_args --flag3 x --flag1 zzz - flag1 passed zzz - flag3 passed x - - $ ./test_args --flag2 asdf - flag2 passed asdf - -We're brushing up against the outer limits of bash here. My prejudice is that you usually shouldn't go this far with bash, because its limitations will come sharply into focus if you try to do too-involved scripting. Instead, use a more friendly language. In Perl, for example, the array containing inputs is @ARGV; in Python, it's sys.argv. Let's compare these common scripting languages: - -| ----- | -| **Bash** | **Perl** | **Python** | **Description** | -| $0 | $0 | sys.argv[0] | Name of Script Itself | -| $* | | | String Containing All Input Arguments | -| ("$@") | @ARGV | sys.argv | Array or List Containing All Input Arguments [1] | -| $1 | $ARGV[0] | sys.argv[1] | First Argument | -| $2 | $ARGV[1] | sys.argv[2] | Second Argument | - -Perl has a [Getopt][109] package that is convenient for reading arguments, and Python has an even better one called [argparse][110]. Their functionality is infinitely nicer than bash's, so steer clear of bash if you're going for a script with lots of options. - -* * * - - -[1] The distinction between $* and $@ is knotty. Dive into these subtleties [on Stackoverflow][111] ↑ - -## Multi-Line Comments, Multi-Line Strings in Bash - -Let's continue in the realm of scripting. You can do a multi-line comment in bash with an if statement: - - # multi-line comment - if false; then - echo hello - echo hello - echo hello - fi - -(Yes, this is a bit of a hack!) - -Multi-line strings are handy for many things. For example, if you want a help section for your script, you can do it like this: - - cat <<_EOF_ - - Usage: - - $0 --flag1 STRING [--flag2 STRING] [--flag3 STRING] - - Required Arguments: - - --flag1 STRING This argument does this - - Options: - - --flag2 STRING This argument does that - --flag3 STRING This argument does another thing - - _EOF_ - -How does this syntax work? Everything between the __EOF__ tags comprises the string and is printed. This is called a [_Here Document][112]_. Read The Linux Documentation Project's discussion of Here Documents [here][113]. - -## Source and Export - -_Question_: If we create some variables in a script and exit, what happens to those variables? Do they disappear? The answer is, yes, they do. Let's make a script called _test_src.sh_ such that: - - $ cat ./test_src.sh - #!/bin/bash - - myvariable=54 - echo $myvariable - -If we run it and then check what happened to the variable on our command line, we get: - - $ ./test_src.sh - 54 - $ echo $myvariable - -The variable is undefined. The command [source][114] is for solving this problem. If we want the variable to persist, we run: - - $ source ./test_src.sh - 54 - $ echo $myvariable - 54 - -and—voilà!—our variable exists in the shell. An equivalent syntax for sourcing uses a dot: - - $ . ./test_src.sh # this is the same as "source ./test_src.sh" - 54 - -But now observe the following. We'll make a new script, _test_src_2.sh_, such that: - - $ cat ./test_src_2.sh - #!/bin/bash - - echo $myvariable - -This script is also looking for _$myvariable_. Running it, we get: - - $ ./test_src_2.sh - -Nothing! So _$myvariable_ is defined in the shell but, if we run another script, its existence is unknown. Let's amend our original script to add in an export: - - $ cat ./test_src.sh - #!/bin/bash - - export myvariable=54 # export this variable - echo $myvariable - -Now what happens? - - $ ./test_src.sh - 54 - $ ./test_src_2.sh - -Still nothing! Why? Because we didn't source _test_src.sh_. Trying again: - - $ source ./test_src.sh - 54 - $ ./test_src_2.sh - 54 - -So, at last, we see how to do this. If we want access on the shell to a variable which is defined inside a script, we must source that script. If we want _other_ scripts to have access to that variable, we must source plus export. - -## Dotfiles (_.bashrc_ and _.bash_profile_) - -Dotfiles are simply files that begin with a dot. We can make a test one as follows: - - $ touch .test - -Such a file will be invisible in the GUI and you won't see it with vanilla ls either. (This works the same way for directories.) The only way to see it is to use the list _all_ option: - -ls -al - -or to list it explicitly by name. This is useful for files that you generally want to keep hidden from the user or discourage tinkering with. - -Many programs, such as bash, [Vim][55], and [Git][92], are highly configurable. Each uses dotfiles to let the user add functionality, change options, switch key bindings, etc. For example, here are some of the dotfiles files each program employs: - -* bash - _.bashrc_ -* vim - _.vimrc_ -* git - _.gitconfig_ -The most famous dotfile in my circle is _.bashrc_ which resides in HOME and configures your bash. Actually, let me retract that: let's say _.bash_profile_ instead of _.bashrc_ (read about the difference [here][115]). In any case, the idea is that this dotfile gets executed as soon as you open up the terminal and start a new session. It is therefore ideal for setting your PATH and other variables, adding functions ([like this one][116]), creating _aliases_ (discussed below), and doing any other setup related chore. For example, suppose you download a new program into /some/path/to/prog and you want to add it to your PATH. Then in your _.bash_profile_ you'd add: - - export PATH=/some/path/to/prog:$PATH - -Recalling how export works, this will allow any programs we run on the command line to have access to our amended PATH. Note that we're adding this to the front of our PATH (so, if the program exists in our PATH already, the existing copy will be superseded). Here's an example snippet of my setup file: - - PATH=/apps/python/2.7.6/bin:$PATH # use this version of Python - PATH=/apps/R/3.1.2/bin:$PATH # use this version of R - PATH=/apps/gcc/4.6.0/bin/:$PATH # use this version of gcc - export PATH - -There is much ado about _.bashrc_ (read _.bash_profile_) and it inspired one of the greatest unix blog-post titles of all time: [_Pimp my .bashrc][117]_—although this blogger is only playing with his prompt, as it were. As you go on in unix and add things to your _.bash_profile_, it will evolve into a kind of fingerprint, optimizing bash in your own unique way (and potentially making it difficult for others to use). - -If you have multiple computers, you'll want to recycle much of your program configurations on all of them. My co-worker uses a nice system I've adopted where the local and global aspects of setup are separated. For example, if you wanted to use certain aliases across all your computers, you'd put them in a global settings file. However, changes to your PATH might be different on different machines, so you'd store this in a local settings file. Then any time you change computers you can simply copy the global files and get your familiar setup, saving lots of work. A convenient way to accomplish this goal of a unified shell environment across all the systems you work on is to put your dotfiles on a server, like [GitHub][94] or [Bitbucket][118], you can access from anywhere. This is exactly what I've done and you can [get the up-to-date versions of my dotfiles on GitHub][119]. - -Here's a sketch of how this idea works: in HOME make a _.dotfiles/bash_ directory and populate it with your setup files, using a suffix of either _local_ or _share_: - - $ ls -1 .dotfiles/bash/ - bash_aliases_local - bash_aliases_share - bash_functions_share - bash_inirun_local - bash_paths_local - bash_settings_local - bash_settings_share - bash_welcome_local - bash_welcome_share - -When _.bash_profile_ is called at the startup of your session, it sources all these files: - - # the directory where bash configuration files reside - INIT_DIR="${HOME}/.dotfiles/bash" - - # to make local configurations, add these files into this directory: - # bash_aliases_local - # bash_paths_local - # bash_settings_local - # bash_welcome_local - - # this line, e.g., protects the functionality of rsync by only turning on the below if the shell is in interactive mode - # In particular, rsync fails if things are echo-ed to the terminal - [[ "$-" != *i* ]] && return - - # bash welcome - if [ -e "${INIT_DIR}/bash_welcome_local" ]; then - cat ${INIT_DIR}/bash_welcome_local - elif [ -e "${INIT_DIR}/bash_welcome_share" ]; then - cat ${INIT_DIR}/bash_welcome_share - fi - - #--------------------LOCAL------------------------------ - # aliases local - if [ -e "${INIT_DIR}/bash_aliases_local" ]; then - source "${INIT_DIR}/bash_aliases_local" - echo "bash_aliases_local loaded" - fi - - # settings local - if [ -e "${INIT_DIR}/bash_settings_local" ]; then - source "${INIT_DIR}/bash_settings_local" - echo "bash_settings_local loaded" - fi - - # paths local - if [ -e "${INIT_DIR}/bash_paths_local" ]; then - source "${INIT_DIR}/bash_paths_local" - echo "bash_paths_local loaded" - fi - - #---------------SHARE----------------------------- - # aliases share - if [ -e "${INIT_DIR}/bash_aliases_share" ]; then - source "${INIT_DIR}/bash_aliases_share" - echo "bash_aliases_share loaded" - fi - - # settings share - if [ -e "${INIT_DIR}/bash_settings_share" ]; then - source "${INIT_DIR}/bash_settings_share" - echo "bash_settings_share loaded" - fi - - # functions share - if [ -e "${INIT_DIR}/bash_functions_share" ]; then - source "${INIT_DIR}/bash_functions_share" - echo "bash_functions_share loaded" - fi - -A word of caution: echoing things in your _.bash_profile_, as I'm doing here, can be dangerous and break the functionaly of utilities like scp and rsync. However, we protect against this with the cryptic line near the top. - -Taking care of bash is the hard part. Other programs are less of a chore because, even if you have different programs in your PATH on your home and work computers, you probably want everything else to behave the same. To accomplish this, just drop all your other configuration files into your _.dotfiles_ repository and link to them from your home directory: - - .gitconfig -> .dotfiles/.gitconfig - .vimrc -> .dotfiles/.vimrc - -## Working Faster with Readline Functions and Key Bindings - -If you've started using the terminal extensively, you might find that things are a bit slow. Perhaps you need some long command you wrote yesterday and you don't want to write the damn thing again. Or, if you want to jump to the end of a line, it's tiresome to move the cursor one character at a time. Failure to immediately solve these problems will push your productivity back into the stone age and you may end up swearing off the terminal as a Rube Goldberg-ian dystopia. So—enter keyboard shortcuts! - -The backstory about shortcuts is that there are two massively influential text editors, [Emacs][57] and [Vim][55], whose users—to be overdramatic—are divided into two warring camps. Each program has its own conventions for shortcuts, like jumping words with your cursor, and in bash they're Emacs-flavored by default. But you can toggle between either one: - - $ set -o emacs # Set emacs-style key bindings (this is the default) - $ set -o vi # Set vi-style key bindings - -Although I prefer Vim as a text-editor, I use Emacs key bindings on the command line. The reason is that in Vim there are multiple modes (normal mode, insert mode, command mode). If you want to jump to the front of a line, you have to switch from insert mode to normal mode, which breaks up the flow a little. In Emacs there's no such complication. Emacs commands usually start with the _Control_ key or the _Meta_ key (usually _Esc_). Here are some things you can do: - -* _Cntrl-a_ \- jump cursor to beginning of line -* _Cntrl-e_ \- jump cursor to end of line -* _Cntrl-k_ \- delete to end of line -* _Cntrl-u_ \- delete to beginning of line -* _Cntrl-w_ \- delete back one word -* _Cntrl-y_ \- paste (yank) what was deleted with the above shortcuts -* _Cntrl-r_ \- reverse-search history for a given word -* _Cntrl-c_ \- kill the process running in the foreground; don't execute current line on the command line -* _Cntrl-z_ \- suspend the process running in the foreground -* _Cntrl-l_ \- clear screen. (this has an advantage over the unix command clear in that it works in the Python, MySQL, and other shells) -* _Cntrl-d_ \- [end of transmission][120] (in practice, often synonymous with quit - e.g., exiting the Python or MySQL shells) -* _Cntrl-s_ \- freeze screen -* _Cntrl-q_ \- un-freeze screen -_These are supremely useful!_ I use these numerous times a day. (On the Mac, the first three even work in the Google search bar!) The first bunch of these fall under the umbrella of [_ReadLine Functions][121]_ (read GNU's extensive documentation [here][122]). There are actually tons more, and you can see them all by entering: - - $ bind -P # show all Readline Functions and their key bindings - $ bind -l # show all Readline Functions - -Four of the most excellent Readline Functions are: - -* _forward-word_ \- jump cursor forward a word -* _backward-word_ \- jump cursor backward a word -* _history-search-backward_ \- scroll through your bash history backward -* _history-search-forward_ \- scroll through your bash history forward -For the first two—which are absolutely indispensable—you can use the default Emacs way: -* _Meta-f_ \- jump forward one word -* _Meta-b_ \- jump backward one word -However, reaching for the _Esc_ key is a royal pain in the ass—you have to re-position your hands on the keyboard. This is where _key-binding_ comes into play. Using the command bind, you can map a Readline Function to any key combination you like. Of course, you should be careful not to overwrite pre-existing key bindings that you want to use. I like to map the following keys to these Readline Functions: - -* _Cntrl-forward-arrow_ \- forward-word -* _Cntrl-backward-arrow_ \- backward-word -* _up-arrow_ \- history-search-backward -* _down-arrow_ \- history-search-forward -In my _.bash_profile_ (or, more accurately, in my global bash settings file) I use: - - # make cursor jump over words - bind '"e[5C": forward-word' # control+arrow_right - bind '"e[5D": backward-word' # control+arrow_left - - # make history searchable by entering the beginning of command - # and using up and down keys - bind '"e[A": history-search-backward' # arrow_up - bind '"e[B": history-search-forward' # arrow_down - -(although these may not work universally [1].) How does this cryptic symbology translate into these particular keybindings? There's a neat trick you can use, to be revealed in the next section. - -_Tip_: On Mac, you can move your cursor to any position on the line by holding down _Option_ and clicking your mouse there. I rarely use this, however, because it's faster to make your cursor jump via the keyboard. - -* * * - - -[1] If you have trouble getting this to work on OS's terminal, try [iTerm2][123] instead, as described [here][124] ↑ - -## More on Key Bindings, the ASCII Table, _Control-v_ - -Before we get to the key binding conundrum, let's review [ASCII][125]. This is, simply, a way of mapping every character on your keyboard to a numeric code. As Wikipedia puts it: - -> The American Standard Code for Information Interchange (ASCII) is a character-encoding scheme originally based on the English alphabet that encodes 128 specified characters—the numbers 0-9, the letters a-z and A-Z, some basic punctuation symbols, some control codes that originated with Teletype machines, and a blank space—into the 7-bit binary integers. - -For example, the character _A_ is mapped to the number _65_, while _q_ is _113_. Of special interest are the _control characters_, which are the representations of things that cannot be printed like _return_ or _delete_. Again [from Wikipedia][126], here is the portion of the ASCII table for these control characters: - -| ----- | -| **Binary** | **Oct** | **Dec** | **Hex** | **Abbr** | [**a]** | [**b]** | [**c]** | **Name** | -| 000 0000 | 000 | 0 | 00 | NUL | ␀ | ^@ | - -[1]: http://en.wikipedia.org/wiki/Unix -[2]: http://en.wikipedia.org/wiki/Linux -[3]: http://en.wikipedia.org/wiki/Command-line_interface -[4]: http://en.wikipedia.org/wiki/Terminal_emulator -[5]: http://en.wikipedia.org/wiki/Shell_script -[6]: http://en.wikipedia.org/wiki/Bourne-again_shell -[7]: http://www.oliverelliott.org/static/article/img/terminal_591.png -[8]: http://www.gnu.org/software/coreutils/ -[9]: http://en.wikipedia.org/wiki/GNU_Core_Utilities -[10]: /static/img/letter_600.jpg -[11]: https://class.coursera.org/startup-001 -[12]: http://ss64.com/bash/rsync.html -[13]: http://www.perl.org -[14]: http://www.python.org -[15]: https://www.gnupg.org/index.html -[16]: http://aws.amazon.com/ec2/ -[17]: http://nginx.org/ -[18]: http://en.wikipedia.org/wiki/Graphical_user_interface -[19]: http://www.youtube.com/watch?v=WiX7GTelTPM -[20]: http://upload.wikimedia.org/wikipedia/commons/c/cd/Unix_timeline.en.svg -[21]: /article/computing/ref_unix/ -[22]: http://www.oliverelliott.org/static/article/img/terminal_119.png -[23]: http://en.wikipedia.org/wiki/Cmd.exe -[24]: http://en.wikipedia.org/wiki/Windows_PowerShell -[25]: http://www.putty.org -[26]: https://chrome.google.com/webstore/detail/secure-shell/pnhechapfaindjhompbnflcldabbghjo?hl=en -[27]: http://mobaxterm.mobatek.net -[28]: http://en.wikipedia.org/wiki/Darwin_(operating_system) -[29]: http://aws.amazon.com/free/ -[30]: http://www.ubuntu.com/download -[31]: http://www.linuxmint.com -[32]: https://getfedora.org/ -[33]: http://www.centos.org -[34]: https://www.cygwin.com/ -[35]: /article/computing/tips_mac/#InstalltheGNUCoreutils -[36]: http://www.oliverelliott.org/static/article/img/root_dir_structure.png -[37]: http://en.wikipedia.org/wiki/Home_directory -[38]: http://www.oliverelliott.org/static/article/img/home_dir_structure.png -[39]: http://www.oliverelliott.org/static/article/img/dir_struct_1125.png -[40]: http://www.thegeekstuff.com/2010/09/linux-file-system-structure/ -[41]: http://en.wikipedia.org/wiki/Uniform_resource_locator -[42]: http://www.e-reading.biz/htmbook.php/orelly/unix2.1/lrnunix/ch03_01.htm -[43]: http://ss64.com/bash/ls.html -[44]: http://www.oliverelliott.org/static/article/img/ls.png -[45]: http://www.oliverelliott.org/static/article/img/ls1.png -[46]: http://www.oliverelliott.org/static/article/img/lshl.png -[47]: http://unixhelp.ed.ac.uk/CGI/man-cgi?finger -[48]: http://en.wikipedia.org/wiki/Hidden_file_and_hidden_directory -[49]: http://www.oliverelliott.org/static/article/img/lsal.png -[50]: http://en.wikipedia.org/wiki/Glob_%28programming%29 -[51]: http://macintoshgarden.org/games/prince-of-persia -[52]: http://en.wikipedia.org/wiki/Microsoft_Word -[53]: http://www.youtube.com/watch?v=znlFu_lemsU -[54]: http://en.wikipedia.org/wiki/Grep -[55]: http://www.vim.org -[56]: http://www.nano-editor.org/ -[57]: http://www.gnu.org/software/emacs/ -[58]: /article/computing/wik_vim/ -[59]: http://www.sublimetext.com/ -[60]: http://aquamacs.org/ -[61]: http://www.peterborgapps.com/smultron/ -[62]: http://en.wikipedia.org/wiki/Escape_character -[63]: http://www.oliverelliott.org/static/article/img/bash_prompt_426.png -[64]: http://en.wikipedia.org/wiki/X_Window_System -[65]: http://www.oliverelliott.org/static/article/img/thepath_410.png -[66]: http://en.wikipedia.org/wiki/Symbolic_link -[67]: http://ss64.com/bash/ln.html -[68]: http://www.oliverelliott.org/static/article/img/myscript_634.png -[69]: https://www.talisman.org/~erlkonig/documents/commandname-extensions-considered-harmful.shtml -[70]: http://en.wikipedia.org/wiki/Shebang_(Unix) -[71]: http://en.wikipedia.org/wiki/Software_portability -[72]: http://en.wikipedia.org/wiki/Env -[73]: http://en.wikipedia.org/wiki/Interpreted_language -[74]: http://en.wikipedia.org/wiki/Compiled_language -[75]: http://en.wikipedia.org/wiki/Compiler -[76]: http://gcc.gnu.org -[77]: https://virtualenv.pypa.io/en/latest/ -[78]: http://stackoverflow.com/questions/5725296/difference-between-sh-and-bash -[79]: http://www.cyberciti.biz/tips/how-do-i-find-out-what-shell-im-using.html -[80]: http://en.wikipedia.org/wiki/Z_shell -[81]: http://en.wikipedia.org/wiki/Tcsh -[82]: http://ss64.com/bash/chmod.html -[83]: http://en.wikipedia.org/wiki/Chmod -[84]: http://en.wikipedia.org/wiki/Secure_Shell -[85]: http://www.ss64.com/bash/ssh.html -[86]: /article/computing/tut_unix/#Dotfilesbashrcandbash_profile -[87]: http://en.wikipedia.org/wiki/RSA_(cryptosystem) -[88]: http://en.wikipedia.org/wiki/Public-key_cryptography -[89]: https://github.com/ -[90]: https://help.github.com/articles/generating-ssh-keys/ -[91]: /article/computing/tips_mac/#sshintoYourMac -[92]: http://git-scm.com/ -[93]: /article/computing/wik_git/ -[94]: https://github.com -[95]: /static/article/example/myscript.html -[96]: http://en.wikipedia.org/wiki/Standard_streams -[97]: http://www.oliverelliott.org/static/article/img/Stdstreams-notitle.svg.png -[98]: http://bash.cyberciti.biz/guide/The_case_statement -[99]: http://en.wikipedia.org/wiki/Array_data_structure -[100]: http://en.wikipedia.org/wiki/Hash_table -[101]: http://www.amazon.com/The-Tiger-Shark-Empirical-Wave-Particle/dp/0521358922 -[102]: http://www.tldp.org/LDP/abs/html/fto.html -[103]: http://tldp.org/LDP/abs/html/exit-status.html -[104]: http://www.oliverelliott.org/static/article/img/lsdirtree_234.jpg -[105]: http://ss64.com/bash/gzip.html -[106]: http://en.wikipedia.org/wiki/Control_flow -[107]: http://wiki.bash-hackers.org/howto/getopts_tutorial -[108]: /static/article/example/test_args.html -[109]: http://perldoc.perl.org/Getopt/Long.html -[110]: https://docs.python.org/2/howto/argparse.html -[111]: http://stackoverflow.com/questions/12314451/accessing-bash-command-line-args-vs -[112]: http://en.wikipedia.org/wiki/Here_document -[113]: http://www.tldp.org/LDP/abs/html/here-docs.html -[114]: http://ss64.com/bash/source.html -[115]: http://www.joshstaiger.org/archives/2005/07/bash_profile_vs.html -[116]: http://www.virtualblueness.net/linux-gazette/109/marinov.html -[117]: http://zxvf-linux.blogspot.com/2013/05/pimp-my-bashrc.html -[118]: https://bitbucket.org -[119]: https://github.com/gitliver/.dotfiles -[120]: http://en.wikipedia.org/wiki/End-of-transmission_character -[121]: http://en.wikipedia.org/wiki/GNU_Readline -[122]: http://tiswww.case.edu/php/chet/readline/readline.html -[123]: http://iterm2.com/ -[124]: /article/computing/tips_mac/#InstalliTerm2 -[125]: http://en.wikipedia.org/wiki/ASCII -[126]: http://en.wikipedia.org/wiki/ASCII#ASCII_control_characters diff --git a/saved-articles/beautiful islands of the south pacific.txt b/saved-articles/beautiful islands of the south pacific.txt deleted file mode 100644 index f3f974b..0000000 --- a/saved-articles/beautiful islands of the south pacific.txt +++ /dev/null @@ -1,215 +0,0 @@ ---- -title: Beautiful Islands of the South Pacific -date: 2010-02-09T19:33:18Z -source: http://kathika.com/destinations/20091202-007408/ -tags: travel, islands - ---- - -Whether it’s the rich blue waters, clean -[beaches](/web/20100106230059/http://kathika.com/tag/beaches/), or -vibrant coral reefs the islands of the south pacific are becoming a -popular tourist destination for many of today’s modern travelers. Sit -back as we take you on a pictorial visit to 12 of the areas most popular -islands. - -[](/web/20100106230059/http://kathika.com/wp-content/uploads/bora-bora-tensaibuta186-ll.jpg "Bora Bora-tensaibuta186")[](/web/20100106230059/http://creativecommons.org/licenses/by/2.0/ "Attribution License") -Photo credit: -[tensaibuta](/web/20100106230059/http://www.flickr.com/photos/97657657@N00/2092792187/ "name") - -Bora Bora - -Bora Bora is a tiny island located in the Pacific Ocean, part of the -Society Islands of French Polynesia. Only about 8,800 people call Bora -Bora home, while many thousands more visit the island and its many -resorts in order to experience the great natural beauty of Bora Bora’s -pristine beaches and crystal clear water. Because of its connection to -[France](/web/20100106230059/http://kathika.com/tag/france/), many of -the island’s citizens speak French and Tahitian, but those who are deal -with tourists do understand English. - -[](/web/20100106230059/http://kathika.com/wp-content/uploads/p1040781.jpg-richard-gifford386-ll.jpg "P1040781.JPG-Richard Gifford386")[](/web/20100106230059/http://creativecommons.org/licenses/by/2.0/ "Attribution License") -Photo credit: [Richard -Gifford](/web/20100106230059/http://www.flickr.com/photos/rgifford/260587788/ "name") - -Fiji - -Fiji is an island nation located in the South Pacific Ocean, a political -grouping of about 322 islands and 522 islets. Out of these islands, only -106 of them are permanently inhabited. Viti Levu and Vanua Levu are the -primary population centers, the both of them supporting 87 percent of -the population. Because of its rich culture and beautiful beaches, this -exotic island nation has become extremely popular as the ideal vacation -and honeymoon destination for many tourists. - -[](/web/20100106230059/http://kathika.com/wp-content/uploads/ocean-color-diversity---huahine-landscape-tiarescott44-ll.jpg "Ocean Color Diversity - Huahine Landscape-tiarescott44")[](/web/20100106230059/http://creativecommons.org/licenses/by/2.0/ "Attribution License") -Photo credit: -[tiarescott](/web/20100106230059/http://www.flickr.com/photos/tiarescott/43401833/ "name") - -Ocean Color Diversity - -Huahine is an island located among the Society Islands, a French -Polynesian archipelago. The island is relatively small, with a length of -about 9.9 miles and a maximum width of 8.1 miles. Two main islands make -up Huahine, both of them surrounding a fringing coral reef. Only a few -hundred yards of water separate the two islands with a sandspit -connected them during low tide. - -[](/web/20100106230059/http://kathika.com/wp-content/uploads/tahiti-beach---huahine---tahiti-pictures-tiarescott449-ll.jpg "Tahiti Beach - Huahine - Tahiti Pictures-tiarescott449")[](/web/20100106230059/http://creativecommons.org/licenses/by/2.0/ "Attribution License") -Photo credit: -[tiarescott](/web/20100106230059/http://www.flickr.com/photos/tiarescott/33505225/ "name") - -Tahiti Beach - Huahine - -Tahiti is the largest island located in Windward group of islands of -French Polynesia. As of August 2007, the island was home to around -180,000 people, making it the most populous island in French Polynesia -by quite a distance. While Tahiti has a very rich culture and history, -thanks to its unique blend of indigenous and French cultures, the island -is most well known internationally because of its popularity as a -vacation destination. - -[](/web/20100106230059/http://kathika.com/wp-content/uploads/tikehau-archipel-des-tuamotus-benoit-mahe858-ll.jpg "Tikehau, archipel des Tuamotus-Benoit Mahe858")[](/web/20100106230059/http://creativecommons.org/licenses/by/2.0/ "Attribution License") -Photo credit: [Benoit -Mahe](/web/20100106230059/http://www.flickr.com/photos/loulou/249803558/ "name") - -Tikehau, archipel des Tuamotus - -Tikehau is a coral atoll located in the Tuamotus Archipelago. -Specifically, Tikehau belongs to the Palliser subgroup of islands, which -is the westernmost group in the Tuamotus. A continuous coral reef -surrounds the lagoon, which contains a great density of fish and other -marine life. About 400 people inhabit the island, supported by a healthy -amount of coconut palms. Tourists resorts also dot the coasts of -Tikehau. - -[](/web/20100106230059/http://kathika.com/wp-content/uploads/the-island-of-moorea-eatatmarks906-ll.jpg "The island of Moorea-eatatmarks906")[](/web/20100106230059/http://creativecommons.org/licenses/by/2.0/ "Attribution License") -Photo credit: -[eatatmarks](/web/20100106230059/http://www.flickr.com/photos/markjtaylor/2821984266/ "name") - -The island of Moorea - -Moorea Island is part of the Society Islands archipelago, which is in -turn located in French Polynesia. Located about nine miles northwest of -Tahiti, Moorea is home to about 17,000 residents. Moorea is a popular -tourist destination, especially for those who are already visiting -French Polynesia, and it has become especially associated with -honeymoons. One can find ads for vacations to Moorea in many wedding -magazines. - -[](/web/20100106230059/http://kathika.com/wp-content/uploads/kia-ora-rangiroa-tensaibuta384-ll.jpg "Kia Ora, Rangiroa-tensaibuta384")[](/web/20100106230059/http://creativecommons.org/licenses/by/2.0/ "Attribution License") -Photo credit: -[tensaibuta](/web/20100106230059/http://www.flickr.com/photos/97657657@N00/3123351125/ "name") - -Kia Ora, Rangiroa - -Rangiroa is the largest atoll in the Tuamotus, which is a chain of -atolls that runs throughout French Polynesia, and it is one of the -largest in the world. About 250 islands, islets, and sandbars make up -the atoll, which is home to about 2,334 inhabitants. Avatoru is the -chief town of the Rangiroa. It is home to several administrative -offices, a post office, many churches and a small airport. - -[](/web/20100106230059/http://kathika.com/wp-content/uploads/vavau-yxo181-ll.jpg "Vavau-YXO181")[](/web/20100106230059/http://creativecommons.org/licenses/by/2.0/ "Attribution License") -Photo credit: -[YXO](/web/20100106230059/http://www.flickr.com/photos/yxo/155767869/ "name") - -Vavau - -Vavau is an island chain consisting of one large island and forty -smaller ones, all of which are a part of Tonga. Vavau’s administrative -center, Neiafu, is the second-largest city in Tonga and it sits at what -is known as one of the best harbor in the entire world. Because of its -excellent harbor and abundant untouched seas, Vavau has gained -recognition as one of the world’s prime fishing spots. - -[](/web/20100106230059/http://kathika.com/wp-content/uploads/raiatea---motu-oatara-othanga443-ll.jpg "Raiatea - Motu Oatara-othanga443")[](/web/20100106230059/http://creativecommons.org/licenses/by/2.0/ "Attribution License") -Photo credit: -[othanga](/web/20100106230059/http://www.flickr.com/photos/24324534@N02/2839561498/ "name") - -Raiatea - Motu Oatara - -Only second to Tahiti in size, Raiatea is the second-largest island in -the Society Islands group of French Polynesia. The most important town -in Raiatea is Utuora, which also serves as the administrative center for -the Leeward Islands group. Raiatea, which is enclosed by a coral reef, -is home to about 12,000 residents and is known as the center of -Polynesia. - -[](/web/20100106230059/http://kathika.com/wp-content/uploads/round-efate-vanuatu-trip-26-nov.-2006---nguna-from-paonangisi-beach-phillipc242-ll.jpg "Round Efate (Vanuatu) trip, 26 Nov. 2006 - Nguna from Paonangisi Beach-PhillipC242")[](/web/20100106230059/http://creativecommons.org/licenses/by/2.0/ "Attribution License") -Photo credit: -[PhillipC](/web/20100106230059/http://www.flickr.com/photos/flissphil/311762598/ "name") - -Vanuatu - -Vanuatu is an island nation located in the South Pacific Ocean, one that -was first inhabited by Melanesian people and then subsequently settled -by Europeans in the 18th century. Vanuatu would change hands between -several European powers before finally establishing independence in 1980 -following a successful independence movement. About 222,000 people live -in Vanuatu, with thousands more people visiting the exotic island locale -each year. - -[](/web/20100106230059/http://kathika.com/wp-content/uploads/rockpools-and-cliff-at-niue-piawaugh65-ll.jpg "Rockpools and cliff at Niue-piawaugh65")[](/web/20100106230059/http://creativecommons.org/licenses/by/2.0/ "Attribution License") -Photo credit: -[piawaugh](/web/20100106230059/http://www.flickr.com/photos/piawaugh/2762569296/ "name") - -Rockpools and cliff at Niue - -Niue, often referred to as “the rock of Polynesia,” is an island nation -located in the South Pacific Ocean. While Niue governs itself, it has a -free association with [New -Zealand](/web/20100106230059/http://kathika.com/new-zealand) and does -not actually have sovereignty, with Queen Elizabeth II serving as the -head of state. The country is in the process of refining its tourism -sector after having identified tourism as one of the best ways for the -country to create revenue. - -[](/web/20100106230059/http://kathika.com/wp-content/uploads/awesome-moment-tata_aka_t588-ll.jpg "Awesome Moment-tata_aka_T588")[](/web/20100106230059/http://creativecommons.org/licenses/by/2.0/ "Attribution License") -Photo credit: -[tata\_aka\_T](/web/20100106230059/http://www.flickr.com/photos/12453467@N00/3824885116/ "name") - -Yap Island - -Yap is an island located in the Caroline Islands grouping, which is in -turn situated in the western Pacific Ocean. While many other neighboring -islands have begun to take on European attitudes, Yap has managed to -retain much of its indigenous culture. About 6,300 live in Yap, -according to a 2003 estimate, while tourism brings thousands more -visitors to Yap each year. - diff --git a/saved-articles/central asia salon literary guide to the world.txt b/saved-articles/central asia salon literary guide to the world.txt deleted file mode 100644 index 40ac88e..0000000 --- a/saved-articles/central asia salon literary guide to the world.txt +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: Destination: Central Asia - Salon.com -date: 2006-06-23T15:26:30Z -source: http://www.salon.com/books/literary_guide/2006/06/22/central_asia/ -tags: magazine, guide, travel, reviews - ---- - -Within the nations collectively known as "the 'stans" one can sense the still-cooling results of numerous historical collisions, not all of them figurative, seeing that an active fault line runs straight through the region. (Uzbekistan's capital, Tashkent, was earthquake-flattened as recently as 1966.) Central Asia is where Europe abuts Asia and Christianity smashes against Islam, where Alexander the Great was stopped dead in his tracks and Genghis Khan and Tamerlane staged their conquests of the known world. The region received one of its first known English visitors in the 1500s, and his subsequent report was measured in its enthusiasm: "These merchants are so beggarly and poor, and bring so little quantity of wares … that there is no hope of any good trade there to be had." - -Variously ruled by Arab conquistadors, Mongol horsemen, Persian meddlers, a series of cruel but sometimes enlightened despots, czarist Russia and, finally — and most disastrously — by the Soviet Union, Central Asia is the geographical equivalent of an oft-forwarded parcel bearing the traces of each of its temporary holder's stamps. It is home to despairingly vast deserts, several large inland seas, endless steppes, beautiful mountainous valleys, and some of the world's least inspiring governments. Its religion is predominantly Islam, but a softer, less ideological incarnation than that which is typically encountered in Arab states, and its culture is a curious agglutination of Soviet, Turkic, Persian and Mongol influences. - -A fine place to begin one's encounter with the diverse and often disquieting literature of Central Asia would be Chingiz Aitmatov's "The Day Lasts More Than a Hundred Years." An ethnic Kyrgyz who writes in Russian, Aitmatov is the only Central Asian fiction writer to have been widely translated in the West. After years of working as an apparatchik in the Soviet government and achieving small prominence as a story writer and playwright, Aitmatov came to world renown with the publication of his first novel in 1980, which broke the gait of lockstep Soviet literary circles to a degree not seen since Alexander Solzhenitsyn's "One Day in the Life of Ivan Denisovich" in 1963. Aitmatov's father was executed during Stalin's purges, and Aitmatov's greatest book is simultaneously a cautious but devastating critique of Stalinism, a fable of a Kazakh man attempting to bury a friend according to Muslim etiquette, and a deeply weird sci-fi novel replete with space stations and what Russian speakers refer to as inoplanetyane (that is, extraterrestrials). - -* * * - -* * * - -Solzhenitsyn's "Cancer Ward" (1968) is partially based on its author's own doubly ill-fated experience of recovering from skin cancer at the tail end of his eight-year sentence to the gulag. (Solzhenitsyn's crime was criticizing Stalin in a letter to a friend during World War II, when he was an officer in the Red Army.) Primarily the story of an unjustly exiled man named Oleg Kostoglotov, "Cancer Ward" is a multicharacter and often bitterly funny study of the lives of exiled Soviets, from still-proud Stalinists to confused men and women unsure of their actual crime to the nurses and doctors who do their best (which is not much) to help "cure" them. It reads as if "M*A*S*H*" had been crossbred with "Darkness at Noon." The unnamed setting of "Cancer Ward" is Tashkent, long considered one of the region's more cosmopolitan cities, but currently an open sore of dictatorial policies and suppurating unrest. "Cancer Ward's" unforgettably bleak final scene takes place in Tashkent's zoo, which was apparently as upsetting in the 1950s as it remains today. - -A more recent novel about Central Asia is Robert Rosenberg's "This Is Not Civilization," published in 2004, which contains what is surely one of contemporary fiction's most beguiling opening lines: "The idea of using porn films to encourage dairy cows to breed was a poor one." Rosenberg is a former Peace Corps volunteer who served in Kyrgyzstan, which reminds us that the Peace Corps, if nothing else, has midwifed some first-rate American literature with an internationalist bent. While Rosenberg's novel roams widely (from Arizona to Kyrgyzstan to Istanbul), its non-Western characters are frequently the most interesting. The novel's central Kyrgyz character, Anarbek, must cope with his daughter's culturally sanctioned "wife-napping" (whereby a young girl is literally stolen from her home and forced to marry) as well as the ruination of his cheese factory. The delights of this novel are primarily its sense of humor and humanely old-fashioned empathy for all. By its final page, one cannot be at all sure which civilization it is that the novel's title piquantly condemns. All of them, probably. - -There are many fine histories detailing the 19th century clash of Russia and Great Britain's imperial ambitions in Central Asia, but the best remains Peter Hopkirk's 1992 book "The Great Game: The Struggle for Empire in Central Asia" (2002). The title refers to what has been called a "Victorian-era Cold War" that, despite both parties' best efforts, never resulted in a direct military confrontation. "Great Game" was coined by one of its central players, an intransigently Christian British officer named Arthur Connolly, whose ghastly fate is one of many that Hopkirk details. While the agents of Europe's two most powerful nations vexed and stymied one another from Kandahar to Khiva, the region's local people watched and waited and occasionally acted, such as during the British retreat from Kabul in 1841. The result was the single largest military disaster in Great Britain's history and, in Hopkirk's retelling, a story as hypnotically monstrous as 9/11 footage. This book is required reading for a general understanding of how modern Central Asia took uncertain shape. - -Fears of militant Islam in the former Soviet republics of Central Asia are, in the popular media, almost always overstated. Seven decades of Soviet atheism have left their mark, and state-appointed religious leaders are still widely derided as "red mullahs." That said, Central Asia has pockets of religious unrest, which ruthless governmental crackdowns, especially in Uzbekistan, have predictably made worse. Ahmed Rashid's "Jihad: The Rise of Militant Islam in Central Asia" is the best primer on this troubling aspect of contemporary Central Asia, and is worth reading if only for a brief story Rashid describes in his preface. While in Dushanbe, Tajikistan, one year into its 5-year-long civil war, Rashid enjoys "a leisurely Sunday lunch" with a Tajik poet, a novelist and a journalist — "the cream of Tajikistan's liberal intelligentsia." Suddenly Rashid is startled by the sound of nearby gunfire. The poet, the novelist and the journalist, to Rashid's shock, "suddenly pulled concealed pistols out of their pockets and fired back." To draw too many conclusions from an anecdote mired in such direly specific circumstances would be unwise, but it does suggest the heat packed by the best of Central Asia's literature — not all of it figurative. - -[ ][1] - -[1]: http://www.salon.com/2006/06/22/central_asia/ diff --git a/saved-articles/decline in style zacalo public square.txt b/saved-articles/decline in style zacalo public square.txt deleted file mode 100644 index e497e5a..0000000 --- a/saved-articles/decline in style zacalo public square.txt +++ /dev/null @@ -1,54 +0,0 @@ ---- -title: Zócalo Public Square :: Decline in Style -date: 2012-01-28T14:25:27Z -source: http://zocalopublicsquare.org/thepublicsquare/2011/11/27/decline-in-style/read/inside-out/ -tags: - ---- - -![Argentina presidential reelection celebration][1] - -**by Gabriel Saez** - -_Tony Soprano: It's good to be in something from the ground floor. I came too late for that, I know. But lately I'm getting the feeling that I came in at the end. The best is over._ - -Dr. Melfi: Many Americans, I think, feel that way. - -For more than a decade I had heard about how great _The Sopranos_ was. So earlier this year I finally decided to give it a try. And how shocked I was when, just four minutes into the pilot, and right after Soprano and Melfi shook hands for the first time, the above exchange took place! It sounded so familiar … I mean, this is a guy of Italian descent complaining about how the present, although richer and more comfortable, is nevertheless worse than the past. It was the incarnation of the Argentinian spirit. - -Growing up in Argentina entails continually practicing historical revisionism and indulging in nostalgia for our bygone greatness. Actually, it is more like nostalgia for the greatness we feel entitled to, but were somehow deprived of. Argentina's psyche resides permanently in the land of "should have, would have, could have." If neighboring Brazil is the perennial country of the future, Argentina is the country of the perennially golden past-we are obsessed with looking back to a time (say, a century ago) when our GDP was comparable to that of European powers, and scratching our heads as to how we could have blown it ever since. If only … - -Argentina is the dean of the club of nations utterly obsessed with their decline, so it is our distinct pleasure to welcome the United States to our dour fraternity-go ahead and take your place, there, alongside France. - -Welcome, but brace yourself for lots of snide comments. I for one am sick and tired of hearing: "Japan is an example of how much you can do with so little; Argentina is the other way around." Then there is the equally annoying: "Australia is what Argentina could have been." Brazilians, meanwhile, make me cringe with their subtle invitation for Argentina to become "their Canada." - -But tough global times make Argentina's experience seem, well, universally relevant. That's why our current president takes advantage of every opportunity she has to preach to the rest of the world-and, more importantly, to the rest of the G20 countries-about the "Argentinian model." It is not a model of development, that's for sure. It is a model of resilience. - -You see, Argentina is a part of Europe in exile-or on probation. So when we look at Greece, we smile. We immediately and instinctively know what all that mess is about. We look at the Spanish _indignados_ as brethren, since we are the perennially outraged. And "Occupy Wall Street" seems to us to be the Hollywood version of "_Que se vayan todos_," the movement that unseated President De La Rua a decade ago, and allowed our country the rare privilege of having five presidents in one week. See, America, you still have a ways to go. - -And don't worry, obsessive decline isn't all bad. It's a bonanza for booksellers, shrinks, and pessimistic political analysts. It turns taxi drivers into philosophers. It seems to do wonders, as well, for red meat-and-wine consumption, not to mention late-night, angst-ridden café conversations. Buenos Aires does late and angst like no one else. Maybe Kansas will see the emergence of its own tango-like melancholic dance. - -But for you Americans to join the club of the obsessively declining nations, there is still one thing you must do. Senseless wars, reckless fiscal policy, and cultural decay aren't enough. You must also shed your quintessentially American core belief that you can reinvent yourselves. Yes, you must give up what Roberto Unger and Cornel West have described as "the American religion of possibility." This idea that it is the natural order of things for your democracy to fulfill an ideal, and for individuals to become fulfilled themselves? Forget it if you know what's good for you. It is an obstacle to the full enjoyment of decline that comes with resigned fatalism. - -That American religion of possibility is what those of us elsewhere have most admired about the United States throughout history. Now that this faith is in retreat in your country, Argentinians can appreciate how it evokes the sense of possibility that once upon a time also drew millions of immigrants to our shores. We're so wrapped up in our historical drama, it is difficult to ascertain how great we really were, or for how long. But people from all over the world did come here in search of happiness and fulfillment. Most of their children still love this land. Our neighbors like us more than they are ready to admit. Maybe, someday, we can still achieve greatness, or at least an equilibrium where the humor and angst involving our "might have beens" will become more folkloric, and less wrenchingly poignant. - -It isn't quite reinventing the notion of reinvention, which may still be within America's grasp, but it might be enough to make the Tony Soprano in all of us feel a bit better. - -_**Gabriel Saez** is a legislative advisor to a member of Argentina's National Congress. _ -_ -*Photo courtesy of [CateIncBA][2]._ - - - - -### Post navigation - -![][3][ Prev][4] ![][5][ Next][6] - - -[1]: http://www.zocalopublicsquare.org/wp-content/uploads/2012/09/Argentina_Style_Decline-600x399.jpg -[2]: http://www.flickr.com/photos/cateincba/6275524212/ -[3]: http://www.zocalopublicsquare.org/wp-content/themes/zocalo/images/icon_prev.png -[4]: http://www.zocalopublicsquare.org/2011/09/15/ah-nolds-love-boat/ideas/inside-out/ -[5]: http://www.zocalopublicsquare.org/wp-content/themes/zocalo/images/icon_next.png -[6]: http://www.zocalopublicsquare.org/2012/05/10/basques-capture-mothers-cup/ideas/inside-out/ diff --git a/saved-articles/destination new guinea salon literary guide to the world.txt b/saved-articles/destination new guinea salon literary guide to the world.txt deleted file mode 100644 index 84d75f4..0000000 --- a/saved-articles/destination new guinea salon literary guide to the world.txt +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: Destination: New Guinea - Salon.com -date: 2006-07-31T23:55:34Z -source: http://www.salon.com/books/literary_guide/2006/07/20/new_guinea/index_np.html -tags: travel, guide, reviews, magazine - ---- - -New Guinea is the second largest island in the world (after Greenland) and the focus of a lot of Western dreaming. This has to do with how relatively unknown the area has been. Contact between native and Westerner took place mostly in the 20th century — much later than in other parts of the colonial world. It is said that more languages are spoken in New Guinea than anywhere else — possibly several hundred — and that the island's inner valleys with their birds of paradise are still not fully explored — though apparently all the cannibals have now learned to eat other things. - -Besides the mainland — the western half of which is part of Indonesia, while the east is the Independent Nation of Papua New Guinea — New Guinea also includes hundreds of islands that lie offshore, including the Trobriands. The legendary anthropologist Bronislaw Malinowski traveled there twice during World War I to research the lives of the islanders. Today, his book "A Diary in the Strict Sense of the Term" would be a blog, but in his own day Malinowski never even thought to make public his private reflections on life in New Guinea. In 1967, 25 years after his death, his widow published his book — and it caused a scandal. No surprise there. "Diary" offers a frank account of Malinowski's boredom with his work, his anger toward the subjects of his research, and his lust for colonial and indigenous women alike. - -Malinowski's book is the expression of a turbulent mind, an elite and daring traveler in a far place, trying to get serious work done. He must fight his own lethargy ("I awoke feeling as if just taken down from a cross — just wasnt functioning") and fight his disdain ("I thought about my present attitude toward ethnogr[aphic] work and the natives. My dislike of them, my longing for civilization"). He must fight his desire, too: "I can repress occasional violent whoring impulses by realizing that it would get me nowhere, that even if I possessed women under these conditions, I would merely be sloshing in the mud." - -In his less loathsome moments, Malinowski also recorded stirring descriptions of New Guinea's watery geography, like this section about passing an island by boat: "Two rocks rise up out of the vegetation, like two truncated pillars out of a heap of overgrown ruins. The sea striving, advancing in orderly rows of long smooth waves. I rowed. At moments I didn't know which way to look — at the exquisite silhouette of Gumasila or the vigorous harmonies of Domdom, or the symphony of pastel colors on the distant mountains of the big island — Sarakeikeine. Flights of birds against the clouds dotting them like buckshot." - -* * * - -* * * - -Move ahead 30 years. New Guinea is part of our history now. So try one of the great antiwar books of all time: "Fear Drive My Feet," published in 1959 by the Australian writer Peter Ryan. Ryan wrote the book when he was 21, teaching at the Australian military academy and reflecting on his experiences two and three years before, behind the lines in the war in New Guinea. Ryan originally tried to publish his book in the 1940s but didn't get anywhere. Years later, a houseguest stuck her nose into the manuscript unbidden, and went to work to find it a home. - -There are so many wonderful things about Ryan's book it is hard to know where to start. It's about youth, and the ways that idealism is killed off in war — or lost, as it was for Ryan. It's an outdoors adventure: Told in a straightforward and open manner, Ryan relates an amazing, and, as it turns out, deadly journey from village to village on New Guinea's Huon Peninsula, and then over 12,000-foot mountains, with the Japanese not far behind. - -Ryan describes the place in a wide-eyed and naturalistic manner, as in this moment, about a family whose dying 8-year-old he cannot heal: "The tultul leant forward and picked up a piece of wood from the fire. He blew gently on it till it flamed, and then held it close to the childs head so that I could see better. Death was already in the little black face - -"No one spoke. As I squeezed out the doorway the mother and father looked bewildered and the old woman followed me with her eyes, detesting my interference. I felt angry at my own helplessness… - -"[Sometime later] A piercing, terrible wail shivered through the air from the village. It was like a dog howling, but infinitely tragic. The child was dead … The wailing became general, taken up, swelling and fading, by every voice in the community … it went on all night." - -The book's majesty is as a screed against war brought to a land that did not ask for it. Here, for instance, is Ryan's portrayal of a wounded Japanese prisoner, being revived so as to be interrogated. "As I looked at his face, wasted with fever and suffering, I suddenly felt more akin to him than to the Australians who would not let him die in peace. His eyes, wonderfully large and soft, met mine. In that brief second, I hoped he could read the message in my face." - -For all the destruction it brought, the war helped to internationalize New Guinea, and make way for art lovers, even tourists. One of those who found her way to the island is Samantha Gillison, a Western woman who landed in New Guinea 60 years after Ryan. "The King of America" is Gillison's 2004 fictionalized account of the death of Michael Rockefeller in 1961 in western New Guinea, where he had gone to collect artworks. Her novel's main business is the psychology of American privilege. Delicately, she lays out the anxieties of Stephen Hesse — her Rockefeller — over his background, his isolation and his true self. - -Hesse comes to feel that he can only find himself in New Guinea, and though he dies there, he is right. The yearning discomfort of his spirit is at last comfortable in this wild place. Gillison's sense of village life is unerring. "Only a few old men and women and quiet children lived in the isolated little hamlet," she writes, sketching a group of houses in the mountains that has drawn the attention of Hesse and a colleague. "The fathers, husbands and sons were either living in the men's house down in the village or had been killed in battle. There was something in this corner of the Koa Valley that was impossibly beautiful to Stephen. It was so full of feeling, as lonely as the end of the world … Soft night was slowly pouring in around them, filling up the mountains. They could see a woman in the glow of the flames. She pulled a sweet-potato tuber out of the ashes, banged it on the mud floor to loose the skin from its steaming white meat…" - -On the one hand, this passage is mundane, showing life stripped to ignoble essentials. On the other, it is utterly transcendent, showing the weariness of a Westerner escaping his materialist condition. In the end, this is why travelers have been drawn to the island: to feel, as no other place can inspire, true spiritual conflict. To have visions while stuck in the mud. - -[ ][1] - -[1]: http://www.salon.com/2006/07/20/new_guinea/ diff --git a/saved-articles/git-annex-metadata-views.txt b/saved-articles/git-annex-metadata-views.txt new file mode 100644 index 0000000..7c74a94 --- /dev/null +++ b/saved-articles/git-annex-metadata-views.txt @@ -0,0 +1,211 @@ +Annex Metadata Filtered Views + +Git annex goes mindbogglingly deep. + +I use git annex to manage my photos -- I have tons of RAW photos in +`~/Pictures` that are backed-up to a few git annex special remotes. + +Whenever I'm done editing a set of raw files I go through this song and +dance: + +``` +cd raw +git annex add +git annex copy --to=s3 +git annex copy --to=nas +git commit -m 'Added a bunch of files to s3' +git push origin master +git push origin git-annex +git annex drop +``` +And magically my raw files are available on s3 and my home NAS. + +If I jump over to a different machine I can run: + +``` +cd ~/Pictures +git fetch +git rebase +git annex get [new-raw-file] +``` +Now that raw file is available on my new machine, I can open it in +Darktable, I can do whatever I want to it: it's just a file. + +This is a pretty powerful extension of git. + +While I was reading the [git annex +internals](https://git-annex.branchable.com/internals/) page today I +stumbled across an even more powerful feature: metadata. You can store +and retrieve arbitrary metadata about any git annex file. For instance, +if I wanted to store EXIF info for a particular file, I could do: + +``` +git annex metadata 20140118-BlazeyAndTyler.jpg --set exif="$(exiftool -S \ + -filename \ + -filetypeextensions \ + -make \ + -model \ + -lensid \ + -focallength \ + -fnumber \ + -iso 20140118-BlazeyAndTyler.jpg)" +``` + +And I can drop that file and still retrieve the EXIF data + +``` +$ git annex drop 20140118-BlazeyAndTyler.jpg +drop 20140118-BlazeyAndTyler.jpg (checking tylercipriani-raw...) (checking tylercipriani-raw...) (checking tylercipriani-raw...) ok +(recording state in git...) +$ git annex metadata --get exif !$ +git annex metadata --get exif 20140118-BlazeyAndTyler.jpg +FileName: 20140118-BlazeyAndTyler.jpg +Make: SAMSUNG +Model: SPH-L720 +FocalLength: 4.2 mm +FNumber: 2.2 +ISO: 125 +``` + +This is pretty neat, but it can also be achieved with [git +notes](https://tylercipriani.com/blog/2016/08/26/abusing-git-notes/) so +it's nothing too spectacular. + +But git annex metadata doesn't quite stop there. + +My picture directory is laid out like this: + +``` +Pictures/ +└── 2015 + └── 2015-08-14_Project-name + ├── bin + │ └── convert-and-resize.sh + ├── edit + │ ├── 2015-08-14_Project-name_00001.jpg + │ └── 2015-08-14_Project-name_00002.jpg + └── raw + └── 2015-08-14_Project-name_00001.NEF +``` + +I have directories for each year, under those directories I create +directories that are prefixed with the [ISO +8601](https://xkcd.com/1179/) import date for the photo, some memorable +project name (like `mom-birthday` or `rmnp-hike`). Inside that directory +I have 2 directories: `raw` and `edit`. Inside each one of those +directories, I have photos that are named with ISO 8601 import date, +project name, and 5-digit import number and a file extension -- raw +files go in `raw` and edited/finished files `edit`. + +I got this system from [Riley +Brandt](http://www.rileybrandt.com/lessons/) (I can't recommend the +Open Source Photography Course enough -- it's amazing!) and it's +served me well. I can find stuff! But git annex really expands the +possibilities of this system. + +## Fictional, real-world, totally real actually happening scenario [¶](https://tylercipriani.com/blog/2016/09/28/git-annex-metadata-filtered-views/#fictional-real-world-totally-real-actually-happening-scenario) + +I go to Rocky Mountain National Park (RMNP) multiple times per year. +I've taken a lot of photos there. If I take a trip there in October I +will generally import those photos in October and create +`2015/2015-10-05_RMNP-hike/{raw,edit}`, and then if I go there again +next March I'd create +`2016/2016-03-21_RMNP-daytrip-with-blazey/{raw,edit}`. So if I want to +preview my RMNP edited photos from October I'd go: + +``` +cd 2015/2015-10-05_RMNP-hike/edit +git annex get +geeqie . +``` + +But what happens if I want to see all the photos I've ever taken in +RMNP? I could probably cook up some script to do this. Off the top of my +head I could do something like `find . -iname '*rmnp*' -type l`, but +that would undoubtedly miss some files from a project in RMNP that I +didn't name with the string `rmnp`. Git annex gives me a different +option: metadata tags. + +## Metadata Tags [¶](https://tylercipriani.com/blog/2016/09/28/git-annex-metadata-filtered-views/#metadata-tags) + +Git annex supports a special type of short metadata -- `--tag`. With +`--tag`, you can tag individual files in your repo. + +The WMF reading team offsite in 2016 was partially in RMNP, but I +didn't name any photos `RMNP` because that wasn't the most memorable +bit of information about those photos (`reading-team-offsite` seemed +like a better project name) nor did `RMNP` represent all the photos from +the offsite. I should tag a few of those photos `rmnp` with git annex: + +``` +$ cd ./2016/2016-05-01_wikimedia-reading-offsite/edit/ +$ git annex metadata --tag rmnp Elk.jpg +metadata Elk.jpg + lastchanged=2016-09-29@04-14-44 + tag=rmnp + tag-lastchanged=2016-09-29@04-14-44 +ok +(recording state in git...) +$ git annex metadata --tag rmnp Reading\ folks\ bing\ higher\ up\ than\ it\ looks.jpg +metadata Reading folks bing higher up than it looks.jpg + lastchanged=2016-09-29@04-14-57 + tag=rmnp + tag-lastchanged=2016-09-29@04-14-57 +ok +(recording state in git...) +``` + +Also, when my old roommate came to town we went to RMNP, but I tagged +those photos `cody-family-adventure-time`. So let's tag a few of those +`rmnp`, too: + +``` +$ cd 2015/2016-01-25_cody-family-adventuretime/edit +$ git annex metadata --tag rmnp alberta-falls.jpg +metadata alberta-falls.jpg + lastchanged=2016-09-29@04-17-48 + tag=rmnp + tag-lastchanged=2016-09-29@04-17-48 +ok +(recording state in git...) +``` + +## Metadata views [¶](https://tylercipriani.com/blog/2016/09/28/git-annex-metadata-filtered-views/#metadata-views) + +Now the thing that was really surprising to me, you can filter the whole +pictures directory based on a particular tag with git annex by using a +[metadata driven +view](https://git-annex.branchable.com/tips/metadata_driven_views/). + +``` +$ tree -d -L 1 +. +├── 2011 +├── 2012 +├── 2013 +├── 2014 +├── 2015 +├── 2016 +├── instagram +├── lib +├── lossy +├── nasa +└── Webcam + +$ git annex view tag=rmnp +view (searching...) +Switched to branch 'views/(tag=rmnp)' +ok +$ ls +alberta-falls_%2015%2016-01-25_cody-family-adventuretime%edit%.jpg +Elk_%2016%2016-05-01_wikimedia-reading-offsite%edit%.jpg +Reading folks bing higher up than it looks_%2016%2016-05-01_wikimedia-reading-offsite%edit%.jpg +``` + +I can even filter this view using other tags with +`git annex vfilter tag=whatever`. And I can continue to edit, refine, +and work with the photo files from there. + +This feature absolutely blew my mind -- I dropped what I was doing to +write this -- I'm trying to think of a good way to work it into my +photo workflow diff --git a/saved-articles/ocean wanderers the ultimate resource for pelagic birding enthusiasts.txt b/saved-articles/ocean wanderers the ultimate resource for pelagic birding enthusiasts.txt deleted file mode 100644 index a9901c9..0000000 --- a/saved-articles/ocean wanderers the ultimate resource for pelagic birding enthusiasts.txt +++ /dev/null @@ -1,306 +0,0 @@ ---- -title: The Ultimate Resource for Pelagic Birding Enthusiasts -date: 2006-05-17T22:00:22Z -source: http://www.oceanwanderers.com/index.html -tags: nature, research - ---- - -> > **Welcome to oceanwanderers.com** -**The on-line resource for serious birders and pelagic enthusiasts** - - -**![][1] -Cover photo: **Bottlenose Dolphin springs from the pressure wave behind a whale watching boat off Cape May New Jersey, July 2005. Photographed using a Canon 10D digital camera and 400 mm Canon lens. Copyright Angus Wilson©2005. - -To view previous cover photos, [click here][2]. -[Tell a friend][3] about this site! - -* * * - -**What's new?** - -****************Gull Identification- [Is this the first documented Western Gull for the Atlantic][4]? -![][5] -******** -[Seabird-News][6] \- New global discussion group on Google. ******** - -******Want to keep up with the latest seabird and marine mammal news from around the world? Become a member of [Seabird-News][7], a new free listserve started by Angus Wilson/Oceanwanderers.com. The archive of messages is open to all but you must be a members to be able to share your sightings from seawatches or boat-based excursions. We also encourage posting of conservation or research news. Hopefully, trip leaders and wildlife tour companies will use this list as a venue to announce forthcoming offshore trips or holidays. To sign up for messages, follow this link: [Subscribe to Seabirds-News][8].****** - -********Shorebird Identification- Japanese Shorebird Blog******** - -********![][9] -Interested in Asiatic shorebirds? Check out the [Shore Birds in Japan][10] blog from Nobuhiro Hashimoto;******** a shorebird enthusiast who birds in Osaka and Mie Prefectures. ****He has photographed a large number of really superb photographs ********captured, ************for the most part, by digiscoping. Many of the images focuse on feather detail, providing a valuable reference collection for birders elsewhere. -![][11] **** - -OW Reviews - Prominar TD-1, the new digital camera/telescope combination from Kowa. -![][12] -**********The shape of things to come? Ocean Wanderers reviews the innovative new spotting scope from Kowa featuring a built in digital camera. [Click here for more][13] (Added 24 Oct 2005).****** - -****Seabird Conservation - Sir David Attenborough joins the _Save the Albatross_ Campaign. -**![][14]BirdLife International's [Save the Albatross campaign][15] received a welcome boost today when broadcaster and naturalist Sir David Attenborough and the organisers of the world's premier ocean sailing challenge – The Volvo Ocean Race - announced their support for this important cause. Other notable public figure to support the campaign include HRH Prince of Wales, transatlantic oarsman John Ridgway and record-breaking yachtswoman Dame Ellen MacArthur. ****** - -******Sir David Attenborough said, "_Albatrosses have survived in the harshest marine environments for 50 million years; more than 100 times longer than our own species. However, these magnificent birds are unable to cope with man-made threats, such as longline fishing. Europeans saw their first albatrosses only 500 years ago in the Southern Ocean, but in our fleeting overlap with these birds we are threatening all but two of the world's 21 species with extinction. It is awe-inspiring to think that some of the albatrosses nesting when I started my broadcasting career are still raising young, half a century later. However, with 100,000 of these birds drowning annually on longlines, the chance of an individual albatross surviving to old age now, seems as remote as the ability of many albatross species to exist beyond the end of this century. Albatrosses should be free to circle the globe for millions of years to come – we must stop this needless slaughter now to prevent an entire branch being torn from the evolutionary tree. It is unthinkable that the only record we will have of these birds will be the attempts of broadcasters, like myself, to share the beauty of our natural world._" ****** - -![][16] - -******The above image (******©******Peter Ryan/Save the Albatross Campaign) shows seabirds (White-chinned Petrels, Giant Petrels, Shy Albatross and Yellow-nosed Albatross) killed by a [long-lining][17] vessel during a single trip. The campaign seeks to persuade governments and fishing authorities around the world to pass and enforce laws to protect albatrosses and other seabirds. Simple measures such as flying colored streams behind the boat when actively fishing and using a plastic shoot to drop baited hooks into the water are highly effective in reducing seabird mortality. There really are no excuses for not making this standard practice worldwide (Added 8 Oct 2005).****** - -****Transoceanic Migrations - Interconnections between populations of Great White Sharks. -An article in the journal _Science_ (7 October 2005: 100-103) described recent satellite tagging studies that reveal an unexpectedly long and rapid migration by Great White Sharks across the Indian Ocean. One adult female (nick-named Nicole after Australian actress Nicole Kidman) was tagged on 7 November 2003 by a team headed by Ramon Bonfil, a scientist with the Wildlife Conservation Society. 'Nicole' spent some time in South African waters before embarking on a 6,900 mile journey straight across the Indian Ocean to western Australia. During this time, the shark made a number of deep dives (up to 3,215 feet) but otherwise spent 61% of the time close to the surface. The satellite device detached itself 99 days later (to allow for collection of the stored data) a mile from shore just south of the Exmouth Gulf in Western Australia. Remarkably, the same shark was resighted off South Africa on 20 August 2004. Other animals in the study did not follow the same path but instead moved up and down the East African coast, including Mozambiquan Territorial waters where they are not protected. These results suggest that the different populations of Great White Sharks may be significantly more interconnected that previously thought and that depletion of one stock will therefore directly impact others (Added 8 Oct 2005).**** - -**Whale & Dolphin Watching- CRESLI. -The [Coastal Research and Education Society of Long Island][18] (CRESLI) has been studying marine mammals in the waters off Long Island (principally New York, Rhode Island and Massachucetts) for more than 20 years. At least 25 species of cetaceans have been shown to utilize Long Island's waters at one time or another, including Fin, Humpback, Minke, North Atlantic Right, Sei, Blue, Sperm and Long-finned Pilot whales, as well as Common, Bottlenose, Striped and Atlantic White-sided Dolphins. During mid-summer (July-August) they run trips to the Great South Channel off Cape Cod. This is a very productive area for whales, including remarkable congregations of the endangered North Atlantic Right Whale on migration between summering grounds in the Canadian maritimes (especially the Bay of Fundy) and the warmer waters off Florida/Georgia and parts unknown (Added 17 Sept 2005).** - -**![][19]![][20] -(Left) Great Shearwater off Long Island, NY 5 June 2005, copyright: Arthur Kopelman©2005. (Right) Breaching Humpback Whale, Great South Channel, MA, August 2005, copyright: Arthur Kopelman©2005.** - -**Where to Watch Seabirds - Heritage Expeditions announces 'Cruise for Conservation'. -[Heritage Expeditions][21] announced the inaugural _"Cruise for Conservation"_, a special voyage to the Subantarctic Islands of Campbell, Auckland and Snares. This journey is being run in association with New Zealand's[ Forest and Bird ][22]and the [Department of Conservation][23], supporters of Birdlife International's [Save the Albatross][24] Campaign. The theme of the trip is for passangers to experience an abundance of albatrosses in their natural element, the Southern Ocean and hopefully bring attention to their growing plight. The expedition will be lead by a number of seabird experts and biologists. Importantly, a percentage of the ticket price will be donated to the [International Save the Albatross Campaign][24], to aid research into albatross biology and possible bycatch prevention solutions. The first voyage will take place on 6th to 12th January, 2006. ** - -For a taste of the extraordinary diversity and numbers of seabird that breed on these wonderful islands, check out the [trip report][25] prepared by John Brodie-Good and myself following our own visit (Nov-Dec 2001) to these jewels of the southern ocean. (Added 8 August 2005) - -**Shorebird Identification - Hybrid Sandpiper or Dunlin? -![][26] -On-going discussion of the identity of two interesting sandpipers photographed in May. (Added 8 August 2005) ** - -**Where to Watch Seabirds - Announcing the 'Western Pacific Odyssey'. - -![][27] -Black Petrel. Photographed in the Hauraki Gulf, New Zealand by Chris Collins© 2005, All rights reserved. ****** - -John Brodie-Good ([WildWings/WildOceans][28]) and Rodney Russ ([Heritage Expeditions, NZ][21]) have just announced an exclusive seabirding adventure through the Western Pacific. Sailing from New Zealand, the cruise will thread its way through the numerous islands of the western Pacific to Kagoshima in Japan. For seabird fanatics this is the proverbial 'once in a lifetime' chance to connect with a host of stunning seabirds including the newly rediscovered New Zealand Storm-Petrel and numerous tropical goodies such as Tahiti Petrel, Gould's Petrel, White-necked Petrel, Heinroth's Shearwater, White-throated Storm-Petrel and Grey Ternlet. The trip will culminate with a visit to Torishima, the principal nesting site for Short-tailed Albatross. A number of landings are planned and participants will encounter numerous hard-to-get landbirds including the awesome Kagu on Noumea in New Caledonia. - -Book soon!! Berths are limited and the ship is filling fast!! - -Click here for more information about the cruise and to view a host of seabird photos from the recce trip by Chris Collins and Kaj Kampp (Added 19 July 2005). - -**Shorebird Identification - Hybrid Sandpiper?** - -![][29] -Wayne Richardson found and superbly photographed this puzzling sandpiper on Marco Island, Florida. Superficially resembling a White-rumped Sandpiper several details appear to be a miss. Click the picture to see more images including a flight shot. Dennis Paulson, author of _Shorebirds of North America: The Photographic Guide_ (2005 Princeton University Press) gets the ball rolling with some comments on the possible parents. (Added 17 June 2005) - -**New pelagic Web Site - [Pterodroma Pelagics][30]. -** -![][31] - -Chris Gaskin & Karen Baird operate [Pterodroma Pelagics][30] as a subsidiary of [Kiwi Wildlife Tours NZ][32]. They have been operating pelagics to the Outer Hauraki Gulf (through Kiwi Wildlife Tours NZ) since the end of 2002, when the Gulf's full potential as a world-class pelagic destination was recognized. Their web site is full of useful information for pelagic birders. Without a doubt, a major highlight of any pelagic to the Hauraki Gulf would be an encounter with the newly discovered New Zealand Storm-Petrel. There are a number of photos of the species scattered across the site and a very interesting article on the work of the NEW ZEALAND STORM PETREL WORKING GROUP, which aims to discover and protect the nesting site(s). Also there is a review of recent sightings - a must for anyone planning a trip. (Added 13 April 2005) - -![][33] -New Zealand Storm-Petrel. Photo by Hadoram Shirihai © Tubenoses Project, A & C Black. - -**Books - Coming Soon: A major new field guide to the marine mammals of the world.** - -![][34]Later this year, A&C Black expects to release "_**A Field Guide to the Marine Mammals of the World Whales, Dolphins, Sirenians and Seals**_" by **Hadoram Shirihai** & **Brett Jarrett**. Intended as a true field guide, the book will show important variation (age, sex and geographical variation) and depict similar species together for ease of comparison. Text and maps will be accompanied by ~400 photographs and ~90 plates, all solely geared towards identification. The compact size of the guide (216 x 135 mm) will allow it to fit in a pocket. For the armchair enthusiast, a larger (A4) format edition, using the same plates and text but containing over 1,000 extra photographs is planned for release at a later date. - -High quality photographs are still being sought for last minute inclusion. Species or forms of interest include:** Fur seals and sea lions:** South African, Australian, Antarctic, Subantarctic, Guadalupe, Juan Fernández, New Zealand, South American, Galápagos and Northern Fur Seals; Californian (includes Galápagos, and Japanese), Northern (Steller's), Australian, New Zealand and South American Sea Lions. **True seals:** Bearded, Harbour, Largha (Spotted), Ringed, Caspian, Baikal, Grey, Ribbon, Harp and Hooded Seals; Mediterranean, Hawaiian and West Indian Monk Seals; Southern and Northern Elephant Seals; Weddell, Ross', Crabeater and Leopard Seals. **Baleen whales:** North Atlantic, North Pacific and Southern Right Whales, Bowhead Whale, Pygmy Right Whale, Gray Whale, Humpback, Northern, Dwarf and Antarctic Minke Whales, Bryde's (includes Eden's, and Omura's), Sei, Fin and Blue Whales (includes Northern, Indian Ocean, Pygmy, and Southern Blue Whales). **Sperm whales:** Sperm Whale, Pygmy Sperm Whale, Dwarf Sperm Whale. **Beaked whales:** Cuvier's, Arnoux's, Baird's and Shepherd's (Tasman) Beaked Whales, Tropical, Northern and Southern Bottlenose Whales, Hector's, True's, Gervais', Sowerby's, Gray's, Pygmy, Andrews', Spade-toothed, Hubbs', Gingko-toothed, Stejneger's, Strap-toothed, Blainville's and Perrin's Beaked Whales. **Oceanic and costal dolphins:** Commerson's, Black (Chilean), Haviside's (Heaviside's), Hector's and Rough-toothed Dolphins, Humpback Dolphins (includes Atlantic, Indian, and Pacific), Tucuxi (includes Atlantic Coast and Amazon), Common and Indo-Pacific Bottlenose Dolphins, Pantropical and Atlantic Spotted Dolphins,) Spinner (includes Gray's or Hawaiian and also Eastern and Central American), Clymene and Striped Dolphins, Short-beaked and Long-beaked Common Dolphins (includes D. c. tropicalis, Arabian Common Dolphin), Fraser's and White-beaked Dolphins, Atlantic and Pacific White-sided Dolphins, Dusky (includes South American, South African and New Zealand), Peale's and Hourglass Dolphins, Northern and Southern Right-whale Dolphins, Risso's Dolphin, Melon-headed Whale, Pygmy Killer Whale, False Killer Whale, Killer Whale, Long-finned and Short-finned Pilot Whales, and Irrawaddy Dolphin. **Porpoises:** Finless (includes Indo-Pacific, Chinese Finless Porpoise, and Yangtse), Harbour, Gulf of California, Spectacled, Burmeister's and Dall's Porpoises. **Arctic cetaceans: **Beluga, Narwhal. **River dolphins and Franciscana: **Ganges, Indus, Amazon and Chinese River Dolphins, and Franciscana. **Sirenians:** West Indian, West African and Amazonian Manatees, Dugong. **Arctic animals:** Walrus, Polar Bear. **Otters:** Marine and Sea Otters. A fee is payable for every photo published. The authors prefer to receive material electronically as digital images or scans of slides, and in high resolution suitable for publication. For submission details please contact Hadoram at [msanroman@bluewin.ch][35]. - -**At the Limits of Ocean and Air** \- New York Times Editorial - Published: January 20, 2005 - -**'Most humans have never seen a [gray-headed albatross][36] or, for that matter, any albatross. And for good reason. The gray-headed albatross breeds in grassy tufts set high on the cliffs of remote islands in the seas just north of Antarctica. But the bird's true habitat is the tumultuous air above those seas. In April 1999, scientists fixed tiny locaters to the albatrosses in a cohort breeding on Bird Island, near South Georgia. The data retrieved \- a map of the migratory patterns of 22 birds - will help scientists understand where albatrosses are most likely to cross paths with fishing boats, which often hook and kill the birds with baited hooks floating just under the water's surface. That could make all the difference to this species, which belongs to the most threatened family of birds on the planet.' ** - -'This research also turned up some surprising glimpses of how albatrosses live. It was already known that they could fly at astonishingly high speeds with Antarctic storms at their backs, and scientists had guessed that the birds were capable of spending great lengths of time aloft. But half the birds in this study flew around the world - as much as 14,000 miles - and one of them did so in 46 days. This implies an ability simply to live on the wing, to rest and forage while making constant headway toward the east and, ultimately, their breeding grounds.' - -'As so often happens, the more we come to know about the life of any individual species, the better we understand how extensive the human impact on this planet really is. Few things seem more remote from our daily lives than the peregrinations of an albatross in the southernmost latitudes. And yet those birds, when they feed, are all too often the immediate victims of our appetite for fish.' - -'It's hard to say how many albatrosses are lost to legal and illegal long-lining ships, which trail enormous numbers of baited hooks behind them. But some estimates say they kill as many as 100,000 birds per year. The long-lining fishing fleet is overharvesting the air as well as the sea.' (Added 20 Jan 2005) - -**Waterfowl Identification- Canada/Cackling Goose Split.** - -**![][37]The [Overview of Canada/Cackling Goose Subspecies][38] has been updated in light of the recent decision by the American Ornithologists' Union to separate the commonly recognized subspecies of Canada Goose into two distinct species. Most of the larger forms remain as 'Canada Goose', whereas the smaller ones are lumped together as 'Cackling Goose'. (6 Oct 2004)** - -**Pelagic Bird Conservation- Betting on Albatrosses** - -**_And they're off_......... In an unusual partnership, the world's biggest bookmaker has teamed up with [Conservation Foundation][39] and the Tasmanian State Government to launch the [Ladbrook's Big Bird Race][40] also known as the 'ultimate flutter'. Scientists from the Tasmanian state government tagged 18 juvenile Shy Albatrosses with satellite transmitters and will follow their migration from Pedra Branca, Albatross Island and Mewstone, three islands lying off the Australian state to feeding grounds in the Benguela Current off Southern Africa. Ladbrokes.com will be offering a variety of bets on the 6000 mile 'race' and punters can follow the birds' progress on-line. Birds are sponsored by celebrity 'owners' including Queen Noor of Jordan ('The Ancient Mariner') and Brian May, the former guitarist for Queen ('Rocky').** - -**An estimated 300,000 seabirds die from longlining each year. One goal of the project is to encourage more countries to sign the Agreement for the Conservation of Albatrosses and Petrels (ACAP). ACAP requires signatory states to take specific measures to reduce seabird by-catch from longline fishing and improve the conservation status of the birds. Initially, only Australia, New Zealand, Ecuador, Spain and South Africa signed the agreement. The UK became the sixth country to ratify in April 2004. Ratification by the UK included the Overseas Territories of the Falklands, British Antarctic Territory and South Georgia/South Sandwich Islands but not Tristan da Cunha, which is crucial and which is due to ratify soon. Most pirate fishing vessels are owned by companies based in Taiwan, Spain, Panama, Singapore, South Korea, Japan, China and Equatorial Guinea. (Added 2 July 2004)** - -**New York Rarities- Bar-tailed Godwit** -**** -![][41] On Friday 28 May 2004, Ken and Sue Feustel discovered a nominate [Bar-tailed Godwit][42] on the flat at Mecox Bay, eastern Long Island. If accepted, this will be the sixth record for the state and first in 19 years. For more images of New York State rarities [click here][43]. (Added 29 May 2004).**** - -**Marine Mammals- New Whale Watching Magazine** -**** -Earlier this month, Rachel Saward launched [Whale and Dolphin Magazine][44]. This beautifully produced publication is aimed at whale and dolphin watchers in the United Kingdom but has plenty to interest enthusiasts further afield.**** - -**Pelagic Birding \- New Bay of Biscay Pelagics** - -**Adam Scott Kennedy has launched a new company [Ultimate Pelagics Ltd][45] offering multiday cruises to bird and mammal rich areas within Europe's hotspot for pelagic wildlife. Details of forthcoming trips are available at [www.ultimatepelagics.com][45].** - -**Pelagic Bird Conservation- Remembering Alec Zino** -** -** **Niklas Holmstrom posted the following sad news: "Paul Alexander (Alec) Zino, the Madeiran ornithologist and conservationist, passed away on 3 March 2004, aged 88. He gave his name to endemic species Zino's Petrel (_Pterodroma madeira_), which is near extinction with only 45 known breeding pairs. Subfossil evidence has revealed that the bird was once abundant on the island, but declined when the first settlers arrived in 1419. ** - -**Alec Zino was born at Quinta Margarida on Madeira on February 9 1916 and was educated in England, where he attended St Edmund's College, Ware, Herts, before going on to read Modern Languages at Christ's College, Cambridge. After graduation he returned to Madeira to work in the family property business.** - -You are welcome to read Brian Unwin's tribute to Alec Zino on: <http://madeira.seawatching.net/zino.html> - -**Give Alec a thought when visiting Madeira, Niklas Holmstrom" (6 April 2004)** - -**Pelagic Birding \- Update on New Zealand Storm-Petrel** - -**Seabird fanatics hoping to catch up with the newly rediscovered New Zealand Storm-Petrel can take heart in continuing sightings from the same area. Brent Stephenson reports that a 18 January trip into the Hauraki Gulf managed to attract 11 or more New Zealand Storm-Petrels to the boat and at times they were the most common species on view - not bad going for a bird that was presumed extinct this time last year! Take a look at the [superb photos and full story][46] on the Wrybill Tours site. (Added 21 Jan 2004)** - -****Where to Watch Seabirds - Madiera**** - -![][47]![][48] - -**Swedish birder, Niklas Holmstrm has created a wonderful web site about [birding and seawatching in the Madeira archipelago][49]. Appropriately named 'Birding Madeira', the site contains all the information you need for a trip to this fascinating archipelago situated in the North Atlantic off North Africa. There are sections on 'Travel info', 'Seawatching', 'Trip reports', 'Photo gallery' and selective lists of birds and more. Niklas is an ardent seawatcher and coauthor of [Flight Identification of European Seabirds][50] (Added 21 Jan 2004) ** - -****Frontiers in Bird ID - North American Juncos**** - -****![][51]**** - -**Juncos are a distinctive group of medium-sized Sparrows found throughout North and Central America. Dark-eyed Junco (_Junco hyemalis_) breeds in boreal habitat from Alaska to Newfoundland and winters widely across North America and into Mexico. The 'species' is highly variable in part due to extensive hybridization between different populations. Juncos present many identification and taxonomic challenges, relevant to birders throughout the USA and Canada. OceanWanderers is proud to archive a lively debate from ID-Frontiers Listerserv centered around [the identification of 'Oregon-type' and 'Pink-sided' juncos found in the eastern half of the continent][52]. An additional page features a [collection of odd juncos from the east][53]. (Added 16 Jan 2004).** - -**Pelagic Birding \- New Zealand Storm-Petrel Rediscovered** - -![][54] - -**The world of seabirds is always full of great surprises. Earlier this year (25th January 2003) a group of pelagic birders led by Brent Stephenson and Sav Saville ([Wrybill Birding Tours][55]) were off Whitianga, New Zealand and photographed an unfamiliar storm-petrel (above). Hitting the books, they came to the heady conclusion that this might be the supposedly extinct and certainly very poorly known New Zealand Storm-Petrel _Oceanites maorianus_. Review of the type specimen at the British Museum in Tring by Hadoram Shirihai and Bill Bourne strongly supports this identification. There almost no information on the natural history of this obscure seabird and there are only three specimens in existence, one at Tring and two in Paris. You can learn more about this extraordinary sighting and view Ian Southey's photos of all three specimens by visiting the dedicated page on the [Wrybill Tours][56] web site.** - -**So is the Whitianga bird a one off? The last of its kind? Apparently not! The thrilling news is that Bob Flood and Bryan Thomas, visiting birders from the UK, observed and photographed a flock of New Zealand Storm-Petrels off Little Barrier Island on 17 November 2003\. They saw 10 birds together, with up to 20 different birds visiting their chum slick over a period of 2 hours. An article on this exciting new development, written by Bob Flood, has just been published in [Birding World][57] (December 10th issue). Note that caption to Plates 6 & 7 is incorrect and should read, 'Black-bellied Storm-petrel _Fregetta tropica_ c.40 kms east of Southport, Queensland, Australia, November 15, 2003 (Bryan Thomas ...'. This additional sighting of multiple individuals should provide the necessary incentive for birders to work with the New Zealand Department of Conservation to quickly discover the breeding localities and ensure their protection. (Added 12 Dec 2003).** - -**Pelagic Birding - Birds and marine mammals of the Eastern Tropical Pacific** - -**![][58]** - -**Read the exclusive and richly illustrated web article aptly titled [Petrel Cocktail][59] written and illustrated by Hadoram Shirihai, describing a recent research cruise through the eastern Pacific starting in Costa Rica and landing in Peru by way of the Galapagos Islands. (Added 6 Dec 2003)** - -**Cetacean Taxonomy - Bryde's Whale split into three distinct species. ** - -**A scientific team led by Dr. Tadasu Yamada of Tokyo's National Science Museum used anatomical and molecular criteria to classify nine balaen whale specimens as belonging to a new species.Their results were published in the journal Nature [2003, 20 Nov. 426 (No 6964): 278-281]. ** - -**Named in honor of the late Dr. Hideo Omura, a celebrated Japanese cetacean biologist, the new whale (Balaenoptera omurai) closely resembles Bryde's Whale (B. brydei) and another cryptic species Eden's Whale (B. edeni). The specimens suggest that Omura's Whales occur mainly in the East China Sea and surrounding waters, with a crudely estimated population of at least 1,000. These exciting discoveries illustrate how little we really know about our marine mammals. How these species can be differentiated at sea remains an open and interesting question for whale watchers and scientists alike. (Added 29 Nov 2003)** - -**Tyrant flycatcher Identification \- ['Western Kingbird' from Geneseo in western New York or something more exotic?][60]** - -![][61] - -**This puzzling _Tyrannus_ flycatcher generated much discussion amongst North American birders. Suggestions ranged from Western, Couch's, Tropical and White-throated Flycatchers to hybrid combinations of Western or Couch's with Scissor-tailed Flycatcher. Although the bird has since disappeared, the debate continues (Added 18 Nov 2003)** - -**Seabird Identification - [Unidentified Pterodroma petrel from Maui, Hawaii][62].** - -![][63] - -**Seabird Identification - Exciting news from the Bismarck Archipelago! ** - -**Working on his forthcoming tubenose monograph, Hadoram Shirihai discovered an unknown feeding ground for the [Heinroth's Shearwater][64] (_Puffinus heinrothi_), with c. 100 birds in a small area. He has generously provided OceanWanderers with the stunning shot posted below. Heinroth's Shearwater is one of the world's least known seabirds and Hadoram's photographs are probably the only images ever taken of the bird in the wild. There are a handful of unconfirmed at-sea sightings from this general area. The breeding grounds are not known with certainty, but most likely lie in the Solomons, east of Papua New Guinea. The same trip produced good numbers of Tahiti Petrels, some Beck's Petrels and a suspected Macgillivray's Petrel, among some other good seabirds and marine mammals. ** - -**Watch this space for more photos and an account of his thrilling adventure! (Added 25 Aug 2003)** - -![][65] - -**Seabird Identification - [Southern Mollymawk on Midway Island, Hawaii 8 April 2003 - Salvin's or Buller's?][66]** - -**[Latest Additions to OceanWanderers][67] Last updated 14 August 2003** - -**[Earlier additions to Ocean Wanderers (1999 to 2001)][68]** - -* * * - -**Main Content** - -**Pelagic Birding and Whale Watching** - -**[Seabird News Round Up][69]** -**[Where to Watch Seabirds][70]** -**[Books about Oceanic Birds and Mammals][71] ** -**[Annotated list of the Seabirds of the World][72]** -**[Annotated list of the Marine Mammals of the World][73]** - -**Shorebird identification** - -**[Shorebirds (Waders) of the World][74]** - -**Rarities and Bird Identification** - -**[Topics in Bird Identification][75]** **Last updated 20 Jan 2004** -**[Archive of Bird Photographs][76]** -**[New York State Rarities][43]** -**[Identification Articles in the Literature][77]** -**[Select List of Recommended Web Sites][78]** - -**Important Tip:** To view the many images on this site, set your monitor to 'real colors' or 'thousands of colors'. The pages may load a tiny bit slower but believe me it makes quite a difference to the image quality! - -[About me][79] - -[1]: http://www.oceanwanderers.com/OWCover.10.2005.jpg -[2]: http://www.oceanwanderers.com/OWOldcovers.html -[3]: %3Ca%20href=%22mailto:?subject=Check%20this%20out&body=Thought%20this%20might%20interest%20you:%20http://www.oceanwanderers.com%22%3E -[4]: http://oceanwanderers.com/unidBBGull.html -[5]: http://www.oceanwanderers.com/unidBBG10.jpg -[6]: http://groups.google.com/group/Seabird-News -[7]: http://groups.google.com/group/Seabird-News?lnk=lr -[8]: http://groups.google.com/group/Seabird-News/subscribe -[9]: http://www.oceanwanderers.com/Nobuhiro_Hashimoto1.jpg -[10]: http://shorebirds.exblog.jp/ -[11]: http://www.oceanwanderers.com/Nobuhiro_Hashimoto2.jpg -[12]: http://www.oceanwanderers.com/TD1Scope.6153.jpg -[13]: http://www.oceanwanderers.com/KowaTD1.html -[14]: http://www.oceanwanderers.com/BirdLifeInt.jpg -[15]: http://www.savethealbatross.net/ -[16]: http://www.oceanwanderers.com/ph_14.jpg -[17]: http://www.birdlife.org/action/science/species/seabirds/longlining.html -[18]: http://www.cresli.org/ -[19]: http://www.oceanwanderers.com/Cresli.GtShwater.jpg -[20]: http://www.oceanwanderers.com/GSC05_1_-439.jpg -[21]: http://www.heritage-expeditions.com/Home -[22]: http://www.forestandbird.org.nz/Marine/albatross.asp -[23]: http://www.doc.govt.nz/index.html -[24]: http://www.birdlife.org/action/campaigns/save_the_albatross/index.html -[25]: http://www.oceanwanderers.com/SubAntNZAUS.html -[26]: http://www.oceanwanderers.com/Dunlin24052001cNDvS.jpg -[27]: http://www.oceanwanderers.com/BlackPetrel.1412ss.jpg -[28]: http://www.wildwings.co.uk/ -[29]: http://www.oceanwanderers.com/Peep%20Sp.jpg -[30]: http://www.nzseabirds.com -[31]: http://www.oceanwanderers.com/new_banner.jpg -[32]: http://www.kiwi-wildlife.co.nz -[33]: http://www.oceanwanderers.com/NZSP5.jpg -[34]: http://www.oceanwanderers.com/SouthernRWDolpin.jpg -[35]: mailto:msanroman%40bluewin.ch -[36]: http://www.oceanwanderers.com/Gryh.Alb.html -[37]: http://www.oceanwanderers.com/CaGo.5442.jpg -[38]: http://www.oceanwanderers.com/CAGO.Subspecies.html -[39]: http://www.conservationfoundation.co.uk/html/news/news_albatross.htm -[40]: http://www.ladbrokes.com/bigbirdrace/ -[41]: http://www.oceanwanderers.com/NYBarTGodwit.0032s.jpg -[42]: http://www.oceanwanderers.com/NYBarTailGodwit.html -[43]: http://www.oceanwanderers.com/NYRarities.html -[44]: http://www.wdmag.co.uk/ -[45]: http://www.ultimatepelagics.com -[46]: http://www.wrybill-tours.com/idproblems/stormpet4.htm%20%0D -[47]: http://www.oceanwanderers.com/Bulwers_Petrel.jpg -[48]: http://www.oceanwanderers.com/Corys_Shearwater.jpg -[49]: http://madeira.seawatching.net/ -[50]: http://theflight.seawatching.net/ -[51]: http://www.oceanwanderers.com/Oct2003.jpg -[52]: http://www.oceanwanderers.com/JuncoID.html -[53]: http://www.oceanwanderers.com/JuncoID2.html -[54]: http://www.oceanwanderers.com/nzsp_brent_stephenson.jpg -[55]: http://www.wrybill-tours.com/ -[56]: http://www.wrybill-tours.com/idproblems/stormpet.htm -[57]: http://www.birdingworld.co.uk/ -[58]: http://www.oceanwanderers.com/PC.HS.JuanFernandezPetrel.JPG -[59]: http://www.oceanwanderers.com/PetrelCocktail.html -[60]: http://www.oceanwanderers.com/NYTyrannus.html -[61]: http://www.oceanwanderers.com/kb37sm1.jpg -[62]: http://www.oceanwanderers.com/MauiPetrel.html -[63]: http://www.oceanwanderers.com/MauiPetrel.5.JPG -[64]: http://www.oceanwanderers.com/HeinrothShear.html -[65]: http://www.oceanwanderers.com/HS.HeinrothsShear.jpg -[66]: http://www.oceanwanderers.com/MidwayAlbert.html -[67]: http://www.oceanwanderers.com/Stoppress.html -[68]: http://www.oceanwanderers.com/Past%20Additions2001.html -[69]: http://www.oceanwanderers.com/SeabirdNews.index.html -[70]: http://www.oceanwanderers.com/BestPelagics.html -[71]: http://www.oceanwanderers.com/PelagicBooks.html -[72]: http://www.oceanwanderers.com/Seabird.Home.html -[73]: http://www.oceanwanderers.com/Mammals.html -[74]: http://www.oceanwanderers.com/OWShore.html -[75]: http://www.oceanwanderers.com/OWIDT.html -[76]: http://www.oceanwanderers.com/AngusBirdingPhotos.html -[77]: http://www.oceanwanderers.com/AngusBirdingCoolstuff.html -[78]: http://www.oceanwanderers.com/AngusBirdingWebSites.html -[79]: http://www.oceanwanderers.com/AboutAngus.html diff --git a/saved-articles/the hidden wonders of the united states you need to visit.txt b/saved-articles/the hidden wonders of the united states you need to visit.txt deleted file mode 100644 index 3a3b3ba..0000000 --- a/saved-articles/the hidden wonders of the united states you need to visit.txt +++ /dev/null @@ -1,151 +0,0 @@ ---- -title: The Hidden Wonders Of The United States You Need To Visit -date: 2015-04-12T02:25:08Z -source: http://all-that-is-interesting.com/hidden-wonders#23 -tags: luxagraf, travel - ---- - -The Black Hills - -![Black Hills American Wonders][1] - -Native Americans have inhabited the Black Hills of South Dakota since at least 7000 BC. The hills were the site of gold mining and as you might guess, numerous battles between the government and Native Americans. Today, they are an annual gathering place for over 550,000 bikers. Source: [Matador Network][2] - -The Black Hills - -The Black Hills landscape is incredibly complex as well, featuring craggy rocks, grasslands and wet valleys. The environment is home to a wide array of animals, including buffalo, mountain lions and Bighorn Sheep. Source: [Matador Network][2] - -Bombay Hook National Wildlife Refuge - -Migratory birds have a friend in Delaware at the Bombay Hook National Wildlife Refuge. Established in 1937, the 15,978-acre tidal marsh is one of the largest and most pristine expanses in the Mid-Atlantic region. Source: [Stephen L Tabone Nature Photography][3] - -Bombay Hook National Wildlife Refuge - -As high-quality habitats along the Atlantic Flyaway disappear, Bombay Hook has become increasingly important as a stop for migratory birds that travel north to their breeding grounds. Source: [Stephen L Tabone Nature Photography][3] - -Carlsbad Caverns - -Tucked in the Guadalupe Mountains of New Mexico is Carlsbad Caverns National Park, where caverns are king. The park contains 119 limestone caves that were carved out by sulfuric acid. Source: [Matador Network][2] - -Carlsbad Caverns - -The caverns were once a part of a primordial sea that existed 250 million years ago. Bones from ice age animals like giant sloths, lions and camels have been found around the entrances to the caves. Source: [Weird World Facts][4] - -Death Valley - -Though Death Valley is the driest and hottest area in North America, it actually sits over one of the world's largest aquifers. The valley's oldest rocks are over 1.7 billion years old. Source: [Matador Network][2] - -Death Valley - -Death Valley is also known for Racetrack Playa, where rocks seem to move without any intervention from humans or animals. Scientists recently discovered that the rocks don't use magic to move, but rather slide across thin sheets of ice that are pushed by wind whipping through the valley. Source: [Matador Network][2] - -Dinosaur Valley State Park - -Just outside of Fort Worth, Texas is a place where you can actually walk in the footsteps of dinosaurs. Dinosaur Valley State Park actually has fossilized dino prints along the Paluxy River that runs through the park. Source: [Dinosaur Valley State Park][5] - -Dinosaur Valley State Park - -Hiking trails take you back through time on rugged and steep pathways, but at least you're not running from a Tyrannosaurus Rex. Source: [Dinosaur Valley State Park][6] - -Hocking Hills State Park - -Picturesque waterfalls and rocky outcroppings aren't normally associated with Ohio, but they're definitely there. Hocking Hills State Park houses unique geographical features. Source: [Business Insider][7] - -Hocking Hills State Park - -Formed by glacial torrents, the park's rock formations also include deep gorges, a rock shelter and a "devil's bathtub," which is a cool way to describe a creepy hole with water in it. Source: [Bourbon Ridge Retreat][8] - -The Horicon Marsh - -The largest freshwater cattail marsh isn't in Florida or Louisiana, it's actually in Wisconsin. The Horicon Marsh is an important habitat for redheaded ducks, Canadian geese and great blue herons. Over 268 different species of birds have been sighted in the area. Source: [Adkotin][9] - -The Horicon Marsh - -The marshland remained unchanged until the arrival of European settlers, who modified it through draining and hunting. However, after it was deemed a wildlife refuge in 1927, water levels returned and it's once again wild. Source: [Birding is Fun][10] - -Craighead Caverns - -The United States' largest non-subglacial underground lake is located outside of the small town of Sweetwater, Tennessee. The lake is part of an extensive cave system called Craighead Caverns. Source: [Travel Mindset][11] - -Craighead Caverns - -Explorers have mapped 13 acres of water and discovered more cavernous rooms beneath the lake. The Lost Sea is marked by "cave flowers," a rare phenomena that worked to have lake named as a National Landmark. Source: [Lake Scientist][12] - -The Monument Rocks - -These beautiful rock formations aren't in the desert of Arizona, but rather in Kansas, in the middle of grassland. Oh, and they're made out of chalk. Source: [Tourist Destinations][13] - -The Monument Rocks - -The Monument Rocks also have the accolades of being named the first national natural landmark by the Department of the Interior. They rise up 70 feet and are estimated to have been formed 80 million years ago. The formations are important shelters for birds, particularly the American kestrel who hunts across the prairie. Source: [Nature's Arches and Bridges][14] - -Mount Desert Island - -Mount Desert Island looms over the water like a mountain, which is how it got its name. The island only has 10,000 year round residents, but visitors come to see Acadia National Park, which is located on the island. Source: [Matador Network][2] - -Acadia National Park - -The island dates back 550 million years ago when it was just a sea-floor mud deposit, created by volcanic ash. Eventually, the island rose and glaciers eroded its landscape, as visible in the extremely rocky landscape. Source: [Matador Network][2] - -Northern Lights, Alaska - -Alaska is one of the best spots on the planet to see the Northern Lights or the Aurora Borealis. Caused by solar winds, the aurora looks like a rainbow doing yoga as it moves across the sky. Source: [National Geographic][15] - -Northern Lights, Alaska - -The Northern Lights are best observed in the winter when it's darkest in Alaska. The displays take place 60 to 70 miles above the Earth, higher than a plane flies. Source: [National Geographic][16] - -The Okefenokee Swamp - -The Okefenokee Swamp covers 700 square miles in southeastern Georgia and northern Florida. The name comes from the Hitchiti Creek language meaning "Waters Shaking." Source: [Luxagraf][17] - -The Okefenokee Swamp - -The shaking waters could come from the sound of the male alligator as it bellows throughout the swamp. Be prepared for awesome paddling treks through 120 miles of swamp trail, just don't fall in. Source: [Luxagraf][17] - -Painted Hills, Oregon - -One of Oregon's 7 natural wonders are the painted hills near the town of Mitchell. Millions of years of history are exposed in the layered hills of the area like geological water painting. Source: [Love These Pics][18] - -Painted Hills, Oregon - -Many ancient fossils have been discovered in the area, including early horses, camels and rhinos. The red coloring of the formations is due to laterite that was created by floodplain deposits. Source: [Love These Pics][18] - -Palouse Falls - -Washington's Palouse Falls consists of upper falls at a drop of about 20 feet, which lead to the main drop and lower falls around 180 feet high. Rock benches, plunge pools and potholes have imprinted the surrounding landscape. Source: [Matador Network][2] - -Palouse Falls - -Kayaker Tyler Bradt ran the falls setting an unofficial world record for highest waterfall run. Lacking that kind of bravery, most of us just enjoy the pristine beauty of the locale. Source: [Reddit][19] - -Pictured Rocks National Lakeshore - -Pictured Rocks National Lakeshore runs nearly 40 rocky and sandy miles along the Lake Superior shoreline in Michigan. The colorful cliffs have been naturally sculpted into caves, peaks and arches. Source: [Random Space][20] - -Pictured Rocks National Lakeshore - -The colors of the painted rocks come from the large amount of minerals in them. The area contains most of Michigan's waterfalls and makes for great recreational activity or even video production. In 2010, Kid Rock filmed the video for his song Born Free at the lakeshore. If he knows about it, you should too! Source: [Random Space][21] - -[1]: http://all-that-is-interesting.com/wordpress/wp-content/uploads/2015/02/hidden_wonders_black_hills.jpg -[2]: http://matadornetwork.com/ -[3]: http://stevetaboneblog.com/page/13/ -[4]: http://www.weirdworldfacts.com/ -[5]: http://tpwd.texas.gov/ -[6]: http://tpwd.texas.gov/state-parks/dinosaur-valley -[7]: http://www.businessinsider.com/ -[8]: http://bourbonridgeretreat.com/ -[9]: https://adkotin.wordpress.com/ -[10]: http://www.birdingisfun.com/ -[11]: http://www.travelmindset.com -[12]: http://www.lakescientist.com/ -[13]: http://www.tourist-destinations.com/ -[14]: http://arches.marbleart.us/ -[15]: http://environment.nationalgeographic.com/ -[16]: http://ngm.nationalgeographic.com/ngm/photo-contest/2011/entries/80665/view/ -[17]: https://luxagraf.net/ -[18]: http://www.lovethesepics.com/ -[19]: http://www.reddit.com/ -[20]: http://www-personal.umich.edu -[21]: http://www-personal.umich.edu/ diff --git a/saved-articles/thoreau 2.0 xoxo conference talk.txt b/saved-articles/thoreau 2.0 xoxo conference talk.txt deleted file mode 100644 index 6d57994..0000000 --- a/saved-articles/thoreau 2.0 xoxo conference talk.txt +++ /dev/null @@ -1,207 +0,0 @@ ---- -title: Thoreau 2.0 - XOXO Conference Talk -date: 2013-10-19T17:52:16Z -source: https://static.pinboard.in/xoxo_talk_thoreau.htm -tags: life - ---- - -![][1] - -This is a picture of me in 2009, right around when I started [Pinboard][2]. I'm standing on a balcony in Botosani county, Romania, in the poorest county in the European Union. - -I'm smiling in this photo, but in reality I was having a rough time. I was $50k in debt, didn't have a job, and the contract work I depended on for my living was drying up because the world economy had just exploded. And to top it off, the Russians were mad at Romania and had turned off the natural gas pipeline. - -I didn't realize it at the time, but this rainbow was the real deal. Pinboard took off much faster than I expected, and I was quickly able to make it my full-time job. Next month will mark four years since the last time I worked for anyone but myself. - -This was a dream come true for me, and lately I've been giving more and more thought about how to keep the party going. Bookmarking sites seem to have a life expectancy of four years, but I don't want to stop! I want to keep doing this for years. - -So in this kind of reflective spirit, I went back and re-read a book that had a profound effect on me when I was younger, that really lit a fire under me about being self-reliant and living a life on my own terms. I wanted to see if it still had anything to say to me now that I was actually doing it. - -And to my relief, it was even better than I remembered. It turns out there was all kinds of stuff I had missed because I was too young, or too callow, to really understand it. - -![][3] - -[at this point I troll the audience by displaying a huge slide of Ayn Rand] - -So today I want to talk to you about one of my biggest heroes, whose work I think will have a profound impact on anyone trying to create something on their own terms. - -![][4] - -Of course I mean this guy, Henry David Thoreau. - -For those of you who weren't forced to read him in high school, he was a content creator active in the 1850's, a pal of Emerson's, active in the Transcendentalist movement. - -We mainly remember him for an experiment in minimal living he conducted at Walden Pond. In 1845, he built himself a small cabin, moved in, and spent two years living and writing about the experience. - -You'll notice he's got a neckbeard going. Louisa May Alcott said of this neckbeard, - -> It will most assuredly deflect amorous advances and preserve the man's virtue in perpetuity - -And she wasn't wrong. But discovering the contraceptive properties of the neckbeard was not the only thing that put Thoreau ahead of his time. - -The man was a pescatarian, a locavore, probably the first environmentalist, a do-it-yourselfer. He practiced yoga, read Hindu sacred texts, and was a bit of a food faddist. He would frequently walk around in camouflage, since that made it easier to sneak up on wild animals. - -In short, he's the kind of guy who would fit right in in Portland. - -![][5] - -So today I want to rebrand Thoreau for the Internet crowd. I wish to present to you Thoreau 2.0. - -![][6] - -First, though, a word of warning. Thoreau is a wonderful writer and often extremely quotable. But when people are very quotable, it can make it harder to listen to what they actually have to say. - -Walden is a layered work. You can't just go in and strip-mine it for a bunch of Tim Ferriss-style life hacks, or inspirational quotes, without missing the entire point of the book. - -Since we have limited time, though, I've gone and picked out some Tim Ferriss-style lifehacks and inspirational quotes, which I will present as a set of bullet points. - -![][7] - -This is probably Thoreau's most famous quote, "Simplify, Simplify, Simplify" - -![][8] - -I like to paraphrase it as: "Simplify". - -You can see here a facsimile of the cabin Thoreau built for himself. - -I don't have to sell this audience on the virtues of simplicity. Over the last few years, it's been a pleasure to see simplicity in design take over. - -But Thoreau was concerned with a different kind of simplicity. He was obsessed with how complexity can creep into unexpected corners of your life, disguised as necessity. He gives the example of a farmer who convinces himself he must eat meat in order to stay strong. Since meat is expensive, the farmer tills more land in order to afford it. And the harder he works, the hungrier he gets, in a vicious spiral. Meanwhile, the ox that pulls his plow makes do with a vegetable diet, and is stronger than the farmer will ever be. - -I had an interesting run-in with this pattern three years ago. I was leaning over a mop bucket when I saw my telephone come tumbling out of my shirt pocket, in slow motion. Even in 2013, water for some reason is still Kryptonite to electronics, and so of course it died on contact. - -Out of a mixture of curiosity (20%) and laziness (80%) I elected not to replace the phone, and see what would happen. - -I was pleasantly surprised to discover that I could live without it. However, it made it impossible for me to get any kind of notification if something happened to Pinboard. - -After worrying about that for a bit, I realized that I was better off not being wakened in the middle of the night if my site was sick. My users probably didn't want me typing into a root console in sleep-deprived panic. Things could wait until morning. - -So that made life a little less stressful. And then I noticed a follow-on effect. Just knowing that there was nothing that could possibly wake me up in an emergency was making me sleep better. - -So what I had thought was a convenience had actually been the foundation for a little pyramid of anxieties. It made me wonder what other stuff in my life was behaving that way. - -But again — I was too lazy to investigate further. - -I'm not arguing, necessarily, that you all should give up your cell phones. But I'm intrigued by this idea of complexity being something adversarial, that sneaks into your life, like a cockroach, and you have to fight to eradicate. - -![][9] - -Thoreau's great problem was that he was a nature writer and an ecologist working at a time before either of those categories made any sense, partly because he was helping to invent them. To his friends and fellow writers, Thoreau was a disappointment, a talented writer who had chosen to squander his gifts picking huckleberries. - -Towards the end of his life, Thoreau actually got very systematic in the data he collected; he may have known more than any man alive about the ecology of New England. But this meant nothing to anyone but him. Nor was it obvious that his nature writing was on a par with the more 'literary' efforts of his time. - -I mention this because all of us in this room are very lucky to be present at the birth of a new medium. It may not feel new, but it really is early days. We get to define what kind of things it will possible to "be" online. That's an enormous privilege. - -But it comes with a price, a kind of centrifugal force you can feel tugging at your identity, pulling you towards other roles that already exist. It can be uncomfortable not to have a name for the thing you do. You have to make sure you plant your feet, and take care to pick the right spot to stand on. - -![][10] - -Thoreau's first book was a failure. He had it published himself, and at some point his publisher made him come down and fetch the rest of the print run, which was taking up too much space. Thoreau hauled it up to Concord, and told a friend at the time: - -> I have now a library of nearly 900 volumes, over 700 of which I wrote myself. - -But this hurt! He had to go back to work at his father's pencil factory (if you've ever used a Conté crayon, you have Henry David Thoreau to thank, since he re-invented the process for making them). And when that didn't work, he had to pivot to farm labor, and finally surveying. Surveying, at least, let him work outside and in the woods, but he was often working for people who wanted to cut down the forest he spent all his free time in. - -There's a pernicious idea that comes out of startup culture called "fail fast". I've always been a big believer in failing slowly. When you're not in it for the money, success doesn't come to you pre-labeled. It can look just like failure. Chasing money makes it easier, because then you can quantify success unambiguously. Otherwise, you may have a hard time telling the two apart. - -You can work on a lot of projects, but you will only get a couple of opportunities to work on something long-term. So I would say pick those carefully, do things that are intrinsically rewarding, and be very loath to abandon them. And work that day job if you have to! - -![][11] - -The best piece of advice Thoreau ever got was from Emerson, who told him to keep a journal. And Thoreau did, for decades, using it as a personal diary, a record of his botanical and scientific observations, and a kind of staging ground for his serious writing. He would go back and mine it years later for passages to use in his work. - -I don't think everyone needs to keep a literary journal, but I think it's vital to keep a work diary, for three reasons: - -First, because it's the only honest record of what you're thinking at the time. Your memory will lie to you, almost immediately, about what you thought was going to happen on any given day. The only way you can trust it is to write down your state of mind - what you're worried about, what you expect will happen. And then over time you can go back and look for patterns of thought that you might want to fix. Maybe you're always too optimistic, or maybe you choose to work with toxic people, or chronically underestimate what things will cost. Writing it down will help you understand your mental habits, and correct for them. - -Second, a work diary helps you track what you're actually doing. It's easy to get lost in the weeds from day to day, but are you ever spending time working on the things you think are most important? Thoreau was mistrustful of trivia the same way he mistrusted complexity, its capacity to take over our lives and push out what we value. An honest work record will tell you what you actually did, and what you spent your time thinking about. - -Finally, and most importantly, writing things down captures the details that you only glean from experience. The one thing separating me from the high-IQ theoreticians on a message board is the fact that I've actually been running a bookmarking site for four years. Experience is priceless, you can't get it except by doing it, so you want to be sure not to fritter any of it away, and document the details as they happen. - -They can come in useful later in the most surprising circumstances. - -![][12] - -A charming thing about _Walden_ is that Thoreau starts it straight away talking about money. He gives the full cost of building his house (around $24) and the total profit he got from cultivating his bean field, about eight bucks. - -If you're not motivated by money, it's easy to talk sometimes as if money is beneath you. But money is beneath us like the ground is beneath us; if it disappears, down we go. Only rich people have the luxury of not talking about money. - -As independent creators, trying to coax other people into striking out on their own, I think we have to take special care to be direct and honest about the financial aspects of our work. - -In working on Pinboard, I've benefitted from other projects that publish their expenses, especially in areas where these numbers would be really hard to estimate. And I've tried to do the same, sharing a spreadsheet with expenses for each year I've run the site. - -Sharing revenue, of course, is scarier. But I've discovered that the only people who really care are other people trying to start solo projects, and they find it helpful. - -So I'll go first - last year Pinboard earned $181,000. By my calculations, this makes me over 23,000 times as successful as Henry David Thoreau. - -![][13] - -> How does it become a man to behave toward the American government today? I answer, that he cannot without disgrace be associated with it. - -In 1846, Thoreau was thrown into jail for failing to pay his poll tax for six years. This was his protest against the Mexican War, which he saw as an attempt to extend slavery into the Western Territories. - -After leaving jail, Thoreau wrote an essay called "[Resistance to Civil Government][14]", where he tried to reason out what we should do when the government compels us to do something morally wrong. - -It's not our job, Thoreau argues, to fix the world. We may not have the time for that. But we can't cooperate with injustice. If the law compels us to do something wrong, we have to break that law. - -This doctrine of non-cooperation with civil authority would have a powerful effect on Gandhi and Martin Luther King. - -Over the past few months, we've heard some shocking things about the extent to which our government is monitoring our online activity. To me more upsetting than the surveillance is the apparatus of secrecy that has been built around it, to make it impossible to observe, or honestly discuss, the full extent of what is going on. - -I've come to believe that it's time for us to take a stand, and refuse to cooperate with this apparatus of secrecy. We've already seen Lavabit, in an act of great moral courage, throw away ten years of hard work rather than acquiesce to blanket monitoring of its users. But the fact that Ladar Levinson wasn't even able to give the reasons for shutting his project down, that we had to infer them from his silence, demonstrates the problem. - -If anyone is going to refuse to cooperate, it is going to be small independent projects, not large corporations. "The rich man—not to make any invidious comparison—is always sold to the institution which makes him rich". - -Larry Page is not going to go to jail. Marissa Mayer is not going to jail - she's already said she thinks [it's tantamount to treason][15]! The large corporations - whose own business model, after all, is surveillance - have folded their hands and said "we've done everything we can within the law to fight this". - -But Thoreau argued, and I agree, that there is a higher law. - -The reason I think it's vital we act now is that this state of affairs is still shocking, still disturbing. Let it persist and it will become the new normal (in other words, the "old shitty") and anyone trying to fight it is going to be branded a Utopian or hopelessly naive, unable to come to terms with modernity. - -We should commit to giving legal, financial and moral support to anyone who refuses to obey gag order, or publishes a National Security Letter. The secrecy exists because the programs it cloaks can't withstand the light of day. One good, timely push will break them. - -Whether or not you agree with me, I would urge you to read Thoreau's essay, and decide for yourself: where do you draw the line? What will it take to make you stop cooperating? - -![][16] - -You could make a cynical case that Thoreau was a bit of a phony. He conducted his experiment in self-reliance on land lent to him by a friend, limited his stay to two years, and received a stream of visitors the entire time. On Sundays, his mom and sister would come out with a basket of doughnuts and pies, which Thoreau devoured, presumably while they washed his socks and made his bed. - -Even his famous moral stand against the poll tax came after just one night in jail. Under cover of darkness, someone (probably his aunt) came by and paid the tax bill, and then paid it every year from then on. - -So Thoreau had all these people, mostly women, who silently enabled the life he thought he was heroically living for himself. - -But a gentler, more generous way to look at it is this. If you live a life by your own lights, and follow your principles, maybe once in a while someone will come and bring you a basket of donuts. And it's okay to eat the donuts! They're delicious! - -Thoreau said about his two years at Walden: - -> I learned this, at least, by my experiment; that if one advances confidently in the direction of his dreams, and endeavors to live the life which he has imagined, he will meet with a success unexpected in common hours. - -Thoreau wrote this never having tasted any of traditional forms of success. He was thinking of a different, more fundamental kind of success, one that I wish for myself, and earnestly wish for all of you. - -But the greatest lesson, the one thing you should retain from reading Thoreau even if you forget everything else, is this: - -![][17] - -Go outside. It's a beautiful day, it's lunchtime, and Portland is a beautiful city! - -THUNDEROUS, PROLONGED APPLAUSE - -[1]: https://static.pinboard.in/thoreau/tp.001.jpg -[2]: https://pinboard.in -[3]: https://static.pinboard.in/thoreau/tp.002.jpg -[4]: https://static.pinboard.in/thoreau/tp.003.jpg -[5]: https://static.pinboard.in/thoreau/tp.004.jpg -[6]: https://static.pinboard.in/thoreau/tp.005.jpg -[7]: https://static.pinboard.in/thoreau/tp.006.jpg -[8]: https://static.pinboard.in/thoreau/tp.007.jpg -[9]: https://static.pinboard.in/thoreau/tp.008.jpg -[10]: https://static.pinboard.in/thoreau/tp.009.jpg -[11]: https://static.pinboard.in/thoreau/tp.010.jpg -[12]: https://static.pinboard.in/thoreau/tp.011.jpg -[13]: https://static.pinboard.in/thoreau/tp.012.jpg -[14]: http://thoreau.eserver.org/civil.html -[15]: http://www.businessinsider.com/marissa-mayer-its-treason-to-ignore-the-nsa-2013-9 -[16]: https://static.pinboard.in/thoreau/tp.013.jpg -[17]: https://static.pinboard.in/thoreau/tp.014.jpg diff --git a/saved-articles/why ad blocking is not a moral dilemma.txt b/saved-articles/why ad blocking is not a moral dilemma.txt deleted file mode 100644 index 378dfc5..0000000 --- a/saved-articles/why ad blocking is not a moral dilemma.txt +++ /dev/null @@ -1,24 +0,0 @@ ---- -title: Why ad blocking is not a moral dilemma -date: 2012-02-07T14:42:14Z -source: http://www.less-broken.com/blog/2011/11/why-ad-blocking-is-not-moral-dilemma.html -tags: economy, morality - ---- - -# Why ad blocking is not a moral dilemma - -_(These are my own views.) _ - -A common meme doing the rounds these days is that blocking advertisements on sites that depend on it is immoral. I don't believe so mainly because fundamental issues like [user sovereignty and control][1] override anything else, but some take the more practical stance that it hurts site owners. I don't think this view is valid. Here's why. - - -* **Web advertising is a relationship among three entities.** It's not just the user and the site owner who are part of it, it's the advertiser too. -* **Using ad blockers means the user's not interested in advertising in the first place.** Clearly, if she were interested in viewing and clicking on ads, she wouldn't be using an ad blocker. Saying that she should _become_ interested in advertising is not a morally tenable position. -* **Loading ads without being interested in them hurts the advertiser.** Assuming a cost-per-impression model, what would benefit the advertiser more: a thousand impressions to people, all interested in advertising, or ten thousand impressions to people, only 10% interested in them? -* **A user not interested in advertising has to "hurt" either of the other two in the relationship. **Either the site owner is hurt because the user doesn't load advertisements, or the advertiser is hurt because the user loads advertising that she was never interested in in the first place. Deciding which one is hurt can't be left to either the owner or the advertiser because both have vested interests – so it has to be done by the user. I don't think there's a way to morally distinguish between the owner and the advertiser. -* **But what about visual impressions?** The argument here is the mere sight of advertising, without a click-through, builds up brand recognition and is therefore of value to the advertiser. However, using this as the basis for some sort of normative argument ("you should subject yourself to advertising so that advertisers can build brand recognition in you!") is basically advocating mind control, hence this isn't morally tenable either. -**Update:** Speak of the devil. A Blogger message just popped up asking if I'd like to add advertising to my blog. - -[1]: http://www.youtube.com/watch?v=2udd765yVMc - diff --git a/tech/debian 7 digital ocean running slowly.txt b/tech/debian 7 digital ocean running slowly.txt deleted file mode 100644 index 9cb2e14..0000000 --- a/tech/debian 7 digital ocean running slowly.txt +++ /dev/null @@ -1,7 +0,0 @@ -http://unix.stackexchange.com/questions/68597/console-kit-daemon-hogging-cpu-and-ram - -Kill the console-kit-daemon process if it's still running. Remove the file - -/usr/share/dbus-1/system-service/org.freedesktop.ConsoleKit.service - -(or move it to some place where you could restore it, if necessary). Reboot and you will see that console-kit-daemon no longer automatically starts up.
\ No newline at end of file diff --git a/tech/how to secure nginx with let's encrypt.txt b/tech/how to secure nginx with let's encrypt.txt deleted file mode 100644 index c071fd3..0000000 --- a/tech/how to secure nginx with let's encrypt.txt +++ /dev/null @@ -1,282 +0,0 @@ ---- -title: How To Secure Nginx with Let's Encrypt on Ubuntu 14.04 -date: 2016-03-20T02:19:27Z -source: https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-14-04 -tags: linux, luxagraf - ---- - -### Introduction - -Let's Encrypt is a new Certificate Authority (CA) that provides an easy way to obtain and install free TLS/SSL certificates, thereby enabling encrypted HTTPS on web servers. It simplifies the process by providing a software client, `letsencrypt`, that attempts to automate most (if not all) of the required steps. Currently, as Let's Encrypt is still in open beta, the entire process of obtaining and installing a certificate is fully automated only on Apache web servers. However, Let's Encrypt can be used to easily obtain a free SSL certificate, which can be installed manually, regardless of your choice of web server software. - -In this tutorial, we will show you how to use Let's Encrypt to obtain a free SSL certificate and use it with Nginx on Ubuntu 14.04. We will also show you how to automatically renew your SSL certificate. If you're running a different web server, simply follow your web server's documentation to learn how to use the certificate with your setup. - -![Nginx with Let's Encrypt TLS/SSL Certificate and Auto-renewal][1] - -## Prerequisites - -Before following this tutorial, you'll need a few things. - -You should have an Ubuntu 14.04 server with a non-root user who has `sudo` privileges. You can learn how to set up such a user account by following steps 1-3 in our [initial server setup for Ubuntu 14.04 tutorial][2]. - -You must own or control the registered domain name that you wish to use the certificate with. If you do not already have a registered domain name, you may register one with one of the many domain name registrars out there (e.g. Namecheap, GoDaddy, etc.). - -If you haven't already, be sure to create an **A Record** that points your domain to the public IP address of your server. This is required because of how Let's Encrypt validates that you own the domain it is issuing a certificate for. For example, if you want to obtain a certificate for `example.com`, that domain must resolve to your server for the validation process to work. Our setup will use `example.com` and `www.example.com` as the domain names, so **both DNS records are required**. - -Once you have all of the prerequisites out of the way, let's move on to installing the Let's Encrypt client software. - -## Step 1 — Install Let's Encrypt Client - -The first step to using Let's Encrypt to obtain an SSL certificate is to install the `letsencrypt` software on your server. Currently, the best way to install Let's Encrypt is to simply clone it from the official GitHub repository. In the future, it will likely be available via a package manager. - -### Install Git and bc - -Let's install Git and bc now, so we can clone the Let's Encrypt repository. - -Update your server's package manager with this command: - -Then install the `git` and `bc` packages with apt-get: - - * sudo apt-get -y install git bc - -With `git` and `bc` installed, we can easily download `letsencrypt` by cloning the repository from GitHub. - -### Clone Let's Encrypt - -We can now clone the Let's Encrypt repository in `/opt` with this command: - - * sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt - -You should now have a copy of the `letsencrypt` repository in the `/opt/letsencrypt` directory. - -## Step 2 — Obtain a Certificate - -Let's Encrypt provides a variety of ways to obtain SSL certificates, through various plugins. Unlike the Apache plugin, which is covered in [a different tutorial][3], most of the plugins will only help you with obtaining a certificate which you must manually configure your web server to use. Plugins that only obtain certificates, and don't install them, are referred to as "authenticators" because they are used to authenticate whether a server should be issued a certificate. - -We'll show you how to use the **Webroot** plugin to obtain an SSL certificate. - -### How To Use the Webroot Plugin - -The Webroot plugin works by placing a special file in the `/.well-known` directory within your document root, which can be opened (through your web server) by the Let's Encrypt service for validation. Depending on your configuration, you may need to explicitly allow access to the `/.well-known` directory. - -If you haven't installed Nginx yet, do so with this command: - - * sudo apt-get install nginx - -To ensure that the directory is accessible to Let's Encrypt for validation, let's make a quick change to our Nginx configuration. By default, it's located at `/etc/nginx/sites-available/default`. We'll use `nano` to edit it: - - * sudo nano /etc/nginx/sites-available/default - -Inside the server block, add this location block: - -Add to SSL server block - - location ~ /.well-known { - allow all; - } - -You will also want look up what your document root is set to by searching for the `root` directive, as the path is required to use the Webroot plugin. If you're using the default configuration file, the root will be `/usr/share/nginx/html`. - -Save and exit. - -Reload Nginx with this command: - - * sudo service nginx reload - -Now that we know our `webroot-path`, we can use the Webroot plugin to request an SSL certificate with these commands. Here, we are also specifying our domain names with the `-d` option. If you want a single cert to work with multiple domain names (e.g. `example.com` and `www.example.com`), be sure to include all of them. Also, make sure that you replace the highlighted parts with the appropriate webroot path and domain name(s): - - * cd /opt/letsencrypt - - * ./letsencrypt-auto certonly -a webroot --webroot-path=/usr/share/nginx/html -d example.com -d www.example.com - -**Note:** The Let's Encrypt software requires superuser privileges, so you will be required to enter your password if you haven't used `sudo` recently. - -After `letsencrypt` initializes, you will be prompted for some information. The exact prompts may vary depending on if you've used Let's Encrypt before, but we'll step you through the first time. - -At the prompt, enter an email address that will be used for notices and lost key recovery: - -![Email prompt][4] - -Then you must agree to the Let's Encrypt Subscribe Agreement. Select Agree: - -![Let's Encrypt Subscriber's Agreement][5] - -If everything was successful, you should see an output message that looks something like this: - - Output: - - IMPORTANT NOTES: - - If you lose your account credentials, you can recover through - e-mails sent to sammy@digitalocean.com - - Congratulations! Your certificate and chain have been saved at - /etc/letsencrypt/live/example.com/fullchain.pem. Your - cert will expire on 2016-03-15. To obtain a new version of the - certificate in the future, simply run Let's Encrypt again. - - Your account credentials have been saved in your Let's Encrypt - configuration directory at /etc/letsencrypt. You should make a - secure backup of this folder now. This configuration directory will - also contain certificates and private keys obtained by Let's - Encrypt so making regular backups of this folder is ideal. - - If like Let's Encrypt, please consider supporting our work by: - - Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate - Donating to EFF: https://eff.org/donate-le - -You will want to note the path and expiration date of your certificate, which was highlighted in the example output. - -**Firewall Note:** If you receive an error like `Failed to connect to host for DVSNI challenge`, your server's firewall may need to be configured to allow TCP traffic on port `80` and `443`. - -**Note:** If your domain is routing through a DNS service like CloudFlare, you will need to temporarily disable it until you have obtained the certificate. - -### Certificate Files - -After obtaining the cert, you will have the following PEM-encoded files: - -* **cert.pem:** Your domain's certificate -* **chain.pem:** The Let's Encrypt chain certificate -* **fullchain.pem:** `cert.pem` and `chain.pem` combined -* **privkey.pem:** Your certificate's private key - -It's important that you are aware of the location of the certificate files that were just created, so you can use them in your web server configuration. The files themselves are placed in a subdirectory in `/etc/letsencrypt/archive`. However, Let's Encrypt creates symbolic links to the most recent certificate files in the `/etc/letsencrypt/live/your_domain_name` directory. Because the links will always point to the most recent certificate files, this is the path that you should use to refer to your certificate files. - -You can check that the files exist by running this command (substituting in your domain name): - - * sudo ls -l /etc/letsencrypt/live/your_domain_name - -The output should be the four previously mentioned certificate files. In a moment, you will configure your web server to use `fullchain.pem` as the certificate file, and `privkey.pem` as the certificate key file. - -### Generate Strong Diffie-Hellman Group - -To further increase security, you should also generate a strong Diffie-Hellman group. To generate a 2048-bit group, use this command: - - * sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048 - -This may take a few minutes but when it's done you will have a strong DH group at `/etc/ssl/certs/dhparam.pem`. - -## Step 3 — Configure TLS/SSL on Web Server (Nginx) - -Now that you have an SSL certificate, you need to configure your Nginx web server to use it. - -Edit the Nginx configuration that contains your server block. Again, it's at `/etc/nginx/sites-available/default` by default: - - * sudo nano /etc/nginx/sites-available/default - -Find the `server` block. **Comment out** or **delete** the lines that configure this server block to listen on port 80. In the default configuration, these two lines should be deleted: - -Nginx configuration deletions - - listen 80 default_server; - listen [::]:80 default_server ipv6only=on; - -We are going to configure this server block to listen on port 443 with SSL enabled instead. Within your `server {` block, add the following lines but replace all of the instances of `example.com` with your own domain: - -Nginx configuration additions — 1 of 3 - - listen 443 ssl; - - server_name example.com www.example.com; - - ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; - ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; - -This enables your server to use SSL, and tells it to use the Let's Encrypt SSL certificate that we obtained earlier. - -To allow only the most secure SSL protocols and ciphers, and use the strong Diffie-Hellman group we generated, add the following lines to the same server block: - -Nginx configuration additions — 2 of 3 - - ssl_protocols TLSv1 TLSv1.1 TLSv1.2; - ssl_prefer_server_ciphers on; - ssl_dhparam /etc/ssl/certs/dhparam.pem; - ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA'; - ssl_session_timeout 1d; - ssl_session_cache shared:SSL:50m; - ssl_stapling on; - ssl_stapling_verify on; - add_header Strict-Transport-Security max-age=15768000; - -Lastly, outside of the original server block (that is listening on HTTPS, port 443), add this server block to redirect HTTP (port 80) to HTTPS. Be sure to replace the highlighted part with your own domain name: - -Nginx configuration additions — 3 of 3 - - server { - listen 80; - server_name example.com www.example.com; - return 301 https://$host$request_uri; - } - -Save and exit. - -Now put the changes into effect by reloading Nginx: - - * sudo service nginx reload - -The Let's Encrypt TLS/SSL certificate is now in place. At this point, you should test that the TLS/SSL certificate works by visiting your domain via HTTPS in a web browser. - -You can use the Qualys SSL Labs Report to see how your server configuration scores: - - In a web browser: - - https://www.ssllabs.com/ssltest/analyze.html?d=example.com - -This SSL setup should report an **A+** rating. - -## Step 4 — Set Up Auto Renewal - -Let's Encrypt certificates are valid for 90 days, but it's recommended that you renew the certificates every 60 days to allow a margin of error. At the time of this writing, automatic renewal is still not available as a feature of the client itself, but you can manually renew your certificates by running the Let's Encrypt client with the `renew` option. - -To trigger the renewal process for all installed domains, run this command: - - * /opt/letsencrypt/letsencrypt-auto renew - -Because we recently installed the certificate, the command will only check for the expiration date and print a message informing that the certificate is not due to renewal yet. The output should look similar to this: - - Output: - - Checking for new version... - Requesting root privileges to run letsencrypt... - /root/.local/share/letsencrypt/bin/letsencrypt renew - Processing /etc/letsencrypt/renewal/example.com.conf - - The following certs are not due for renewal yet: - /etc/letsencrypt/live/example.com/fullchain.pem (skipped) - No renewals were attempted. - -Notice that if you created a bundled certificate with multiple domains, only the base domain name will be shown in the output, but the renewal should be valid for all domains included in this certificate. - -A practical way to ensure your certificates won't get outdated is to create a cron job that will periodically execute the automatic renewal command for you. Since the renewal first checks for the expiration date and only executes the renewal if the certificate is less than 30 days away from expiration, it is safe to create a cron job that runs every week or even every day, for instance. - -Let's edit the crontab to create a new job that will run the renewal command every week. To edit the crontab for the root user, run: - -Add the following lines: - - crontab entry - - 30 2 * * 1 /opt/letsencrypt/letsencrypt-auto renew >> /var/log/le-renew.log - 35 2 * * 1 /etc/init.d/nginx reload - -Save and exit. This will create a new cron job that will execute the `letsencrypt-auto renew` command every Monday at 2:30 am, and reload Nginx at 2:35am (so the renewed certificate will be used). The output produced by the command will be piped to a log file located at `/var/log/le-renewal.log`. - -For more information on how to create and schedule cron jobs, you can check our [How to Use Cron to Automate Tasks in a VPS][6] guide. - -## Step 5 — Updating the Let's Encrypt Client (optional) - -Whenever new updates are available for the client, you can update your local copy by running a `git pull` from inside the Let's Encrypt directory: - - * cd /opt/letsencrypt - - * sudo git pull - -This will download all recent changes to the repository, updating your client. - -## Conclusion - -That's it! Your web server is now using a free Let's Encrypt TLS/SSL certificate to securely serve HTTPS content. - -[1]: https://assets.digitalocean.com/articles/letsencrypt/nginx-letsencrypt.png -[2]: https://www.digitalocean.com/community/articles/initial-server-setup-with-ubuntu-14-04 -[3]: https://www.digitalocean.com/community/tutorials/how-to-secure-apache-with-let-s-encrypt-on-ubuntu-14-04 -[4]: https://assets.digitalocean.com/articles/letsencrypt/le-email.png -[5]: https://assets.digitalocean.com/articles/letsencrypt/le-agreement.png -[6]: https://www.digitalocean.com/community/tutorials/how-to-use-cron-to-automate-tasks-on-a-vps diff --git a/tech/install tinytinyrss debian 10.txt b/tech/install tinytinyrss debian 10.txt deleted file mode 100644 index 0c65213..0000000 --- a/tech/install tinytinyrss debian 10.txt +++ /dev/null @@ -1,5 +0,0 @@ -ttrss - -php modules debian: - -php-fpm php-xml php-mbstring php-pgsql diff --git a/tech/mount exfat as read write.txt b/tech/mount exfat as read write.txt new file mode 100644 index 0000000..4aa68b1 --- /dev/null +++ b/tech/mount exfat as read write.txt @@ -0,0 +1 @@ +sudo mount -o rw,users,uid=1000,dmask=007,fmask=117 /dev/sdxn /mnt/sd1 # general syntax diff --git a/tech/set up certbot nginx ubuntu.txt b/tech/set up certbot nginx ubuntu.txt deleted file mode 100644 index f13b1e9..0000000 --- a/tech/set up certbot nginx ubuntu.txt +++ /dev/null @@ -1,137 +0,0 @@ -EFF's free certificate service, Certbot, has greatly simplified the task of setting up HTTPS for your websites. The only downside is that the certificates are only good for 90 days. Fortunately renewing is easy, and we can even automate it all with systemd. Here's how to set up Certbot with Nginx *and* make sure your SSL certs renew indefinitely with no input from you. - -This tutorial is aimed at anyone using an Ubuntu 18.04 VPS from cheap hosts like DigitalOcean or [Vultr.com](https://www.vultr.com/?ref=6825229), but should also work for other versions of Ubuntu, Debian, Fedora, CentOS and any other system that uses systemd. The only difference will be the commands you use to install Certbot. See the Certbot site for [instructions](https://certbot.eff.org/) specific to your system. - -Here's how you get Certbot running on Ubuntu 18.04, then we'll dive into setting up automatic renewals via systemd. - -You should not need this with 18.04, but to be on the safe side, make sure you have the `software-properties-common` package installed. - -~~~~console -sudo apt install software-properties-common -~~~~ - -The next part requires that you add a PPA, my least favorite part of Certbot for Ubuntu, as I don't like to rely on PPAs for something as mission critical as my security certificates. Still, as of this writing, there is not a better way. At least go [look at the code](https://launchpad.net/~certbot/+archive/ubuntu/certbot) before you blindly cut and paste. When you're done, here's your cut and paste: - -~~~~console -sudo apt update -sudo add-apt-repository ppa:certbot/certbot -sudo apt update -sudo apt install python-certbot-nginx -~~~~ - -Now you're ready to install some certs. For this part I'm going to show the commands and the output of the commands since the `certbot` command is interactive. Note that the version below will append some lines to your Nginx config file. If you prefer to edit your config file yourself, use this command: `sudo certbot --nginx certonly`, otherwise, here's now to get your certs. - -~~~~console -sudo certbot --nginx - -[sudo] password for $youruser: -Saving debug log to /var/log/letsencrypt/letsencrypt.log -Plugins selected: Authenticator nginx, Installer nginx - -Which names would you like to activate HTTPS for? -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -1: luxagraf.net -2: awstats.luxagraf.net -3: origin.luxagraf.net -4: www.luxagraf.net -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Select the appropriate numbers separated by commas and/or spaces, or leave input blank to select all options shown (Enter 'c' to cancel): 4 -Obtaining a new certificate -Performing the following challenges: -http-01 challenge for www.luxagraf.net -Waiting for verification... -Cleaning up challenges -Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/luxagraf.net.conf - -Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access. -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -1: No redirect - Make no further changes to the webserver configuration. -2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for -new sites, or if you're confident your site works on HTTPS. You can undo this -change by editing your web server's configuration. -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2 - -Traffic on port 80 already redirecting to ssl in /etc/nginx/sites-enabled/luxagraf.net.conf -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Congratulations! You have successfully enabled https://www.luxagraf.net. -You should test your configuration at: https://www.ssllabs.com/ssltest/analyze.html?d=www.luxagraf.net -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -IMPORTANT NOTES: - - Congratulations! Your certificate and chain have been saved at: - /etc/letsencrypt/live/www.luxagraf.net/fullchain.pem - Your key file has been saved at: - /etc/letsencrypt/live/www.luxagraf.net/privkey.pem - Your cert will expire on 2019-01-09. To obtain a new or tweaked - version of this certificate in the future, simply run certbot again - with the "certonly" option. To non-interactively renew *all* of - your certificates, run "certbot renew" - - If you like Certbot, please consider supporting our work by: - Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate - Donating to EFF: https://eff.org/donate-le -~~~~ - -And there you have it, SSL certs for all your domains. - -That's all good and well, but those new certs are only good for 90 days. The odds of you remembering to renew that every 90 days -- even with reminder emails from the EFF -- is near nil. Plus, do you really want to be renewing certs by hand, [like an animal](http://5by5.tv/hypercritical/17)? No, you want to automate everything so you can do better things with your time. - -You could use cron, but the more modern approach would be to create a systemd service and a systemd timer to control when that service runs. - -I highly recommend reading through the Arch Wiki page on [systemd services and timers](https://wiki.archlinux.org/index.php/Systemd/Timers), as well as the [systemd.timer man pages](https://jlk.fjfi.cvut.cz/arch/manpages/man/systemd.timer.5) to get a better understanding of how you can automate other tasks in your system. But for the purposes of this tutorial all you really need to understand is that timers are just like other systemd unit files, but they include a `[Timer]` block which takes parameter for exactly when you want your service file to run. - -Timer files can live right next to your service files in `/etc/systemd/system/`. - -There's no hard and fast rules about naming timers, but it makes sense to use same name as the service file the timer controls, except the timer gets the `.timer` extension. So you'll have two files `myservice.service` and `myservice.timer`. - -Let's start with the service file. I call mine `certbot-renewal`. Open the service file: - -~~~~console -sudo nano /etc/systemd/system/certbot-renewal.service -~~~~ - -This is going to be a super simple service, we'll give it a description and a command to run and that's it: - -~~~~ini -[Unit] -Description=Certbot Renewal - -[Service] -ExecStart=/usr/bin/certbot renew -~~~~ - -Next we need to create a .timer file that will run the certbot.renewal service every day. Create this file: - -~~~~console -sudo nano /etc/systemd/system/certbot-renewal.timer -~~~~ - -And now for the slightly more complex timer: - -~~~~ini -[Unit] -Description=Certbot Renewal Timer - -[Timer] -OnBootSec=500 -OnUnitActiveSec=1d - -[Install] -WantedBy=multi-user.target -~~~~ - -The `[Timer]` directive can take a number of parameters, the ones we've used constitute what's called a monotonic timer, which means they run "after a time span relative to a varying starting point". In other words they're not calendar events like cron. - -Our monotonic timer has two directives, `onBootSec` and `OnUnitActiveSec`. The first should be obvious, our timer will run 500 seconds after the system boots. Why 500? No real reason, I just didn't want to bog down the system at boot. - -The `OnUnitActiveSec` is really what makes this work. This directive measures time relative to when the service that the timer controls was last activated. In our case the `1d` means run the service one day after it last ran. So our timer will run once a day to make sure our scripts stay up to date. - -As a kind of footnote, in systemd parlance calendar-based timers are called realtime timers and can be used to replace cron if you want, though there are some disadvantages, see the Arch Wiki for [a good overview of what you get and what you lose](https://wiki.archlinux.org/index.php/Systemd/Timers#As_a_cron_replacement) if you go that route. - -Okay, the last step for our certbot renewal system is to enable and then start our timer. Note that we don't have to do either to our actual service file because we don't want it active, the timer will control when it runs. - -~~~~console -sudo systemctl enable certbot-renewal.timer -sudo systemctl start certbot-renewal.timer -~~~~ - -Run those commands and you're done. Your timer is now active and your Certbot certificates will automatically renew as long as your server is up and running. diff --git a/tech/set up debian droplet python 3 + gunicorn + supervisor.txt b/tech/set up debian droplet python 3 + gunicorn + supervisor.txt deleted file mode 100644 index 3f199b9..0000000 --- a/tech/set up debian droplet python 3 + gunicorn + supervisor.txt +++ /dev/null @@ -1,166 +0,0 @@ -Set Up Debian Droplet - Python 3 + gunicorn + supervisor - -[reference: -<http://michal.karzynski.pl/blog/2013/06/09/django-nginx-gunicorn-virtualenv-supervisor/> -<http://wiki.nginx.org/HttpHeadersMoreModule#more_clear_input_headers> -<https://github.com/nbs-system/naxsi/wiki/basicsetup> -<http://pillow.readthedocs.org/en/latest/installation.html#linux-installation> -<http://codeinthehole.com/writing/how-to-install-postgis-and-geodjango-on-ubuntu/> -<http://docs.gunicorn.org/en/latest/configure.html> -<udo update-rc.ttp://edvanbeinum.com/how-to-install-and-configure-supervisord/> -] - -If you really want python3.3 you can compile it from scratch. That would eliminate the need to install virtualenv, since it's part of Python as of 3.3. I've gone that route, but for simplicity's sake most of the time I just use python 3.2 which has is available in the debian stable repos. I also grab pip from the repos, though gunicorn and supervisor I install via pip since I want those on a virtualenv-based per-project basis. - -So start with this: - - apt-get install python3.2 python3.2-dev python3-pip - -And then: - - pip-3.2 install virtualenv - -That gets us a nice python3 working setup, though note that you have to call python with python3 and pip-3.2. To cut down on the typing I just make aliases in my .zshrc along the lines of: - - alias p3="python3 " - alias p3p="pip-3.2 " - -Okay so we can use that to setup a working django environment with `virtualenv`. You can use [`virtualenvwrapper`](http://virtualenvwrapper.readthedocs.org/en/latest/) if you like, I find it to be unnecessary. I do something like this: - - mkdir -p apps/mydjangoapp - cd !$ - virtualenv --distribute --python=python3 venv - source venv/bin/activate - -There are few other things that you may want to install before we get around to actually installing stuff with pip. For example if you plan to use memcached you'll want to install pylibmc which needs: - - sudo apt-get install python-dev libmemcached-dev - -**Apparently pylibmc doesn't work with Python3 yet** - -Then I just load everything I need from my requirements.txt file, which lives in the config folder: - - pip install -r config/requirements.txt - -Where did that file come from? Typically I generate it with `pip freeze > requirements.txt` in my local development environment. - -Among the requirements will be gunicorn. If you don't already have a requirements file then you'd just do this: - - pip install gunicorn - -Okay, so we have our sandboxed python3 environment, along with gunicorn to act as a server for our site. In a minute we'll connect Nginx and gunicorn, but first let's make sure our gunicorn server restarts whenever our machine reboots. To do that we'll use `supervisor`. Here's where it gets tricky though, `supervisor` doesn't run under Python 3. It has no problem *managing* python 3 projects, it just doesn't run under python 3 yet. That means we can't just install it using pip3.2. - -We could install it with the system pip, but debian (and ubuntu) have a supervisor repo, so we can just do: - - sudo apt-get install supervisor - -That will install and start supervisor. Let's add an init script so that supervisord starts up should the server need to reboot. so create the file - - /etc/init.d/supervisord - -And grab the appropriate [init script from the supervisor project](https://github.com/Supervisor/initscripts). I use the Debian script from that link. Paste that script into `/etc/init.d/supervisord` and save. Then make it executable: - - sudo chmod +x /etc/init.d/supervisord - -Now, make sure supervisor isn't running: - - supervisorctl shutdown - -And add supervisor to - -With Supervisor installed you can start and watch apps by creating configuration files in the `/etc/supervisor/conf.d` directory. You might do something like this, in, for example, `/etc/supervisor/conf.d/helloworld.conf`: - - [program:helloworld] - command = /home/<username>/apps/mydjangoapp/venv/bin/gunicorn -c /home/<username>/apps/mydjangoapp/config/gunicorn_config.py config.wsgi - directory = /home/<username>/apps/mydjangoapp/ - user = <non-privledged-user> - autostart = true - autorestart = true - stdout_logfile = /var/log/supervisor/helloworld.log - stderr_logfile = /var/log/supervisor/helloworld_err.log - -You'll need to fill in the correct paths based on your server setup, replacing <username> with your username and `mydjangoapp/etc...` with the actual path to the gunicorn app. This also assumes your gunicorn config file lives in `mydjangoapp/config/`. We'll get to that file in a minute. - -First, let's tell supervisor about our new app: - - sudo supervisorctl reread - -You should see a message `helloworld available`. So Supervisor knows about our app, let's actually add it. - - sudo supervisorctl update - -Now you should see a message that says something like `helloworld: added process group`. Supervisor is now aware of our hello world app and will make sure it automatically starts up whenever our server reboots. You can check the status of our gunicorn app with: - - sudo supervisorctl status - -Right now that will generate an error that looks something like this: - - helloworld FATAL can't find command '/home/<username>/apps/mydjangoapp/venv/bin/gunicorn' - - - - - -Now we just need to set up that gunicorn_config.py file we referenced earlier. - -In my setup that file looks like this: - - from os.path import dirname, abspath,join - # get the root folder for this project, which happens to be two folder up - PROJ_ROOT = abspath(dirname(dirname(dirname(__file__))))+'/' - command = join(PROJ_ROOT, "/venv/bin/gunicorn") - pythonpath = PROJ_ROOT - bind = '127.0.0.1:8002' - workers = 3 - log_level = "warning" - error_logfile = "/home/<username>/logs/gunicorn.error.log" - -This is pretty boilerplate, you just need to adjust the paths and it should work. The other thing to note is the line `bind = '127.0.0.1:8002'`. That's the address we'll pass requests to with Nginx. - -Okay, now let's go back to the Nginx tutorial we worked with in the previous part of this series. Here's what this looks like: - - - - # define an upstream server named gunicorn on localhost port 8002 - upstream gunicorn { - server localhost:8002; - } - - server { - listen 80; - server_name mydomain.com; - root /var/www/mydomain.com/; - error_log <path to logs>/mydomain.error.log main; - access_log <path to logs>/mydomain.access.log main; - # See http://wiki.nginx.org/HttpCoreModule#client_max_body_size - client_max_body_size 0; - - # this tries to serve a static file at the requested url - # if no static file is found, it passes the url to gunicorn - try_files $uri @gunicorn; - - # define rules for gunicorn - location @longgunicorn { - # repeated just in case - client_max_body_size 0; - - # proxy to the gunicorn upstream defined above - proxy_pass http://gunicorn; - - # makes sure the URLs don't actually say http://gunicorn - proxy_redirect off; - # If gunicorn takes > 3 minutes to respond, give up - proxy_read_timeout 3m; - - # make sure these HTTP headers are set properly - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - - } - - - } - - - diff --git a/tech/set up geodjango on debian 7 digital ocean.txt b/tech/set up geodjango on debian 7 digital ocean.txt deleted file mode 100644 index 475c6eb..0000000 --- a/tech/set up geodjango on debian 7 digital ocean.txt +++ /dev/null @@ -1,98 +0,0 @@ -the first thing we need is python 3.4+ which I install by hand: - -prereqs: -sudo apt-get install build-essential libncursesw5-dev libssl-dev libgdbm-dev libc6-dev libsqlite3-dev tk-dev libreadline6-dev - -wget https://www.python.org/ftp/python/3.4.1/Python-3.4.1.tgz -tar -xvzf Python-3.4.1.tgz -cd Python-3.4.1 -./configure --prefix=/opt/python3 -make -sudo make altinstall - -Then we need postgres and the geospatial libs: - -apt-get install postgresql postgresql-contrib binutils libproj-dev gdal-bin - - - -postgis 2 i also do from source: - -wget http://download.osgeo.org/postgis/source/postgis-2.1.3.tar.gz -tar -xvzf postgis-2.1.3.tar.gz -cd postgis-2.1.3 - -preqs: - -apt-get install libpq-dev postgresql-server-dev-all libxml2 libgeos-dev libxml2-dev gdal-bin libgdal-dev - -./configure -make -sudo make install - -Then you just need to create a db user and database - -sudo su - postgres -createuser -P -s -e luxagraf - -The with regular user: -createdb luxagraf -U luxagraf -W -hlocalhost -psql -U luxagraf -W -hlocalhost -d luxagraf -and when you're in postgres: -CREATE EXTENSION postgis; - -Then just load the data: -psql -U luxagraf -W -hlocalhost -d luxagraf -f fullbak.sql (or whatever your backupfile is) - - -The last thing is a virtualenv for our project using python3.4 - -#make sure you use the right pip (included with python 3.4+) -sudo /opt/python3/bin/pip3.4 install virtualenv -then cd to proj dir and do: -/opt/python3/bin/virtualenv --distribute --python=/opt/python3/bin/python3-4 venv -then activate and install what you need - ---- -Once that's set up we need to connect gunicorn to nginx via supervisor. - - sudo apt-get install supervisor - -That will install and start supervisor. Let's add an init script so that supervisord starts up should the server need to reboot. so create the file - - /etc/init.d/supervisord - -And grab the appropriate [init script from the supervisor project](https://github.com/Supervisor/initscripts). I use the Debian script from that link. Paste that script into `/etc/init.d/supervisord` and save. Then make it executable: - - sudo chmod +x /etc/init.d/supervisord - -Now, make sure supervisor isn't running: - - supervisorctl shutdown - -And add supervisor to - -With Supervisor installed you can start and watch apps by creating configuration files in the `/etc/supervisor/conf.d` directory. You might do something like this, in, for example, `/etc/supervisor/conf.d/helloworld.conf`: - - [program:helloworld] - command = /home/<username>/apps/mydjangoapp/venv/bin/gunicorn -c /home/<username>/apps/mydjangoapp/config/gunicorn_config.py config.wsgi - directory = /home/<username>/apps/mydjangoapp/ - user = <non-privledged-user> - autostart = true - autorestart = true - stdout_logfile = /var/log/supervisor/helloworld.log - stderr_logfile = /var/log/supervisor/helloworld_err.log - -You'll need to fill in the correct paths based on your server setup, replacing <username> with your username and `mydjangoapp/etc...` with the actual path to the gunicorn app. This also assumes your gunicorn config file lives in `mydjangoapp/config/`. We'll get to that file in a minute. - -First, let's tell supervisor about our new app: - - sudo supervisorctl reread - -You should see a message `helloworld available`. So Supervisor knows about our app, let's actually add it. - - sudo supervisorctl update - -Now you should see a message that says something like `helloworld: added process group`. Supervisor is now aware of our hello world app and will make sure it automatically starts up whenever our server reboots. You can check the status of our gunicorn app with: - - sudo supervisorctl status
\ No newline at end of file diff --git a/tech/set up gitea on ubuntu 18.04.txt b/tech/set up gitea on ubuntu 18.04.txt deleted file mode 100644 index 6d1870b..0000000 --- a/tech/set up gitea on ubuntu 18.04.txt +++ /dev/null @@ -1,233 +0,0 @@ -I've never liked hosting my git repos on someone else's servers. GitHub especially is not a company I'd do business with, ever. I do have a repo or two hosted over at [GitLab](https://gitlab.com/luxagraf) because those are projects I want to be easily available to anyone. But I store almost everything in git -- notes, my whole documents folder, all my code projects, all my writing, pretty much everything is in git -- but I like to keep all that private and on my own server. - -For years I used [Gitlist](http://gitlist.org/) because it was clean, simple, and did 95 percent of what I needed in a web-based interface for my repos. But Gitlist is abandonware at this point and broken if you're using PHP 7.2. There are few forks that [patch it](https://github.com/patrikx3/gitlist), but it's copyrighted to the original dev and I don't want to depend on illegitimate forks for something so critical to my workflow. Then there's self-hosted Gitlab, which I like, but the system requirements are ridiculous. - -Some searching eventually led me to Gitea, which is lightweight, written in Go and has everything I need. - -Here's a quick guide to getting Gitea up and running on your Ubuntu 18.04 -- or similar -- VPS. - -### Set up Gitea - -The first thing we're going to do is isolate Gitea from the rest of our server, running it under a different user seems to be the standard practice. Installing Gitea via the Arch User Repository will create a `git` user, so that's what I used on Ubuntu 18.04 as well. - -Here's a shell command to create a user named `git`: - -~~~~console -sudo adduser --system --shell /bin/bash --group --disabled-password --home /home/git git -~~~~ - -This is pretty much a standard adduser command such as you'd use when setting up a new VPS, the only difference is that we've added the `--disable-password` flag so you can't actually log in with it. While we will use this user to authenticate over SSH, we'll do so with a key, not a password. - -Now we need to grab the latest Gitea binary. At the time of writing that's version 1.5.2, but be sure to check the [Gitea downloads page](https://dl.gitea.io/gitea/) for the latest version and adjust the commands below to work with that version number. Let's download the Gitea binary and then we'll verify the signing key. Verifying keys is very important when working with binaries since you can't see the code behind them[^1]. - -~~~~console -wget -O gitea https://dl.gitea.io/gitea/1.5.2/gitea-1.5.2-linux-amd64 -gpg --keyserver pgp.mit.edu --recv 0x2D9AE806EC1592E2 -wget https://dl.gitea.io/gitea/1.5.2/gitea-1.5.2-linux-amd64.asc -gpg --verify gitea-1.5.2-linux-amd64.asc gitea -~~~~ - -A couple of notes here, GPG should say the keys match, but then it should also warn that "this key is not certified with a trusted signature!" That means, essentially, that this binary could have been signed by anybody. All we know for sure is that wasn't tampered with in transit[^1]. - -Now let's make the binary executable and test it to make sure it's working: - -~~~~console -chmod +x gitea -./gitea web -~~~~ - -You can stop Gitea with `Ctrl+C`. Let's move the binary to a more traditional location: - -~~~~console -sudo cp gitea /usr/local/bin/gitea -~~~~ - -The next thing we're going to do is create all the directories we need. - -~~~~console -sudo mkdir -p /var/lib/gitea/{custom,data,indexers,public,log} -sudo chown git:git /var/lib/gitea/{data,indexers,log} -sudo chmod 750 /var/lib/gitea/{data,indexers,log} -sudo mkdir /etc/gitea -sudo chown root:git /etc/gitea -sudo chmod 770 /etc/gitea -~~~~ - -That last line should make you nervous, that's too permissive for a public directory, but don't worry, as soon as we're done setting up Gitea we'll change the permissions on that directory and the config file inside it. - -Before we do that though let's create a systemd service file to start and stop Gitea. The Gitea project has a service file that will work well for our purposes, so let's grab it, make a couple changes and then we'll add it to our system: - -~~~~console -wget https://raw.githubusercontent.com/go-gitea/gitea/master/contrib/systemd/gitea.service -~~~~ - -Now open that file and uncomment the line `After=postgresql.service` so that Gitea starts after postgresql is running. The resulting config file should look like this: - -~~~~ini -[Unit] -Description=Gitea (Git with a cup of tea) -After=syslog.target -After=network.target -#After=mysqld.service -After=postgresql.service -#After=memcached.service -#After=redis.service - -[Service] -# Modify these two values and uncomment them if you have -# repos with lots of files and get an HTTP error 500 because -# of that -### -#LimitMEMLOCK=infinity -#LimitNOFILE=65535 -RestartSec=2s -Type=simple -User=git -Group=git -WorkingDirectory=/var/lib/gitea/ -ExecStart=/usr/local/bin/gitea web -c /etc/gitea/app.ini -Restart=always -Environment=USER=git HOME=/home/git GITEA_WORK_DIR=/var/lib/gitea -# If you want to bind Gitea to a port below 1024 uncomment -# the two values below -### -#CapabilityBoundingSet=CAP_NET_BIND_SERVICE -#AmbientCapabilities=CAP_NET_BIND_SERVICE - -[Install] -WantedBy=multi-user.target -~~~~ - -Now we need to move the service file to somewhere systemd expects it and then start and enable the service so Gitea will launch automatically when the server boots. - -~~~~console -sudo cp gitea.service /etc/systemd/system/ -sudo systemctl enable gitea -sudo systemctl start gitea -~~~~ - -There you have it, Gitea is installed, running and will automatically start whenever we restart the server. Now we need to set up Postgresql and then Nginx to serve up our Gitea site to the world. Or at least to us. - -### Setup a Postgresql and Nginx - -Gitea needs a database to store all our data in; I use PostgreSQL. You can also use MySQL, but you're on your own there. Install PostgreSQL if you haven't already: - -~~~~console -sudo apt install postgresql -~~~~ - -Now let's create a new user and database for Gitea: - -~~~~console -sudo su postgres -createuser gitea -createdb gitea -O gitea -~~~~ - -Exit the postgres user shell by hitting `Ctrl+D`. - -Now let's set up Nginx to serve our Gitea site. - -~~~~console -sudo apt update -sudo apt install nginx -~~~~ - -For the next part you'll need a domain name. I use a subdomain, git.mydomain.com, but for simplicity sake I'll refer to `mydomain.com` for the rest of this tutorial. Replace `mydomain.com` in all the instructions below with your actual domain name. - -We need to create a config file for our domain. By default Nginx will look for config files in `/etc/nginx/sites-enabled/`, so the config file we'll create is: - -~~~~console -nano /etc/nginx/sites-enabled/mydomain.com.conf -~~~~ - -Here's what that file looks like: - -~~~~nginx -server { - listen 80; - listen [::]:80; - server_name <mydomain.com>; - - - location / { - proxy_pass http://localhost:3000; - } - - proxy_set_header X-Real-IP $remote_addr; -} -~~~~ - -The main line here is the `proxy_pass` bit, which takes all requests and sends it to gitea, which is listening on `localhost:3000` by default. You can change that if you have something else that conflicts with it, but you'll need to change it here and in the service file that we used to start Gitea. - -The last step is to add an SSL cert to our site so we can clone over https (and SSH if you keep reading). I have another tutorial on setting up [Certbot for Nginx on Ubuntu](/src/certbot-nginx-ubuntu-1804). You can use that to get Certbot installed and auto-renewing certs. Then all you need to do is run: - -~~~~console -sudo certbot --nginx -~~~~ - -Select your Gitea domain, follow the prompts and when you're done you'll be ready to set up Gitea. - -### Setting up Gitea - -Point your browser to `https://mydomain.com/install` and go through the Gitea setup process. That screen looks like this, and you can use these values, except for the domain name (and be sure to enter the password you used when we created the `gitea` user for postgresql). - -One note, if you intend your Gitea instance to be for you alone, I strongly recommend you check the "disable self registration" box, which will stop anyone else from being able to sign up. But, turning off registration means you'll need to create an administrator account at the bottom of the page. - -<img src="images/2018/gitea-install_FAW0kIJ.jpg" id="image-1706" class="picwide" /> - -Okay, now that we've got Gitea initialized it's time to go back and change the permissions on those directories that we set up earlier. - -~~~~console -sudo chmod 750 /etc/gitea -sudo chmod 644 /etc/gitea/app.ini -~~~~ -Now you're ready to create your first repo in Gitea. Click the little button next to the repositories menu on the right side of your Gitea dashboard and that'll walk you through creating your first repo. Once that's done you can clone that repo with: - -~~~~console -git clone https://mydomain.com/giteausername/reponame.git -~~~~ - -Now if you have an existing repo that you want to push to your new Gitea repo, just edit the `.git/config` files to make your Gitea repo the new url, e.g.: - -~~~~ini -[remote "origin"] - url = https://mydomain.com/giteausername/reponame.git - fetch = +refs/heads/*:refs/remotes/origin/* -~~~~ - -Now do this: - -~~~~console -git push origin master -~~~~ - -### Setting up SSH - -Working with git over https is pretty good, but I prefer the more secure method of SSH with a key. To get that working we'll need to add our SSH key to Gitea. That means you'll need a GPG key. If you don't have one already, open the terminal on your local machine and issue this command: - -~~~~console -ssh-keygen -o -a 100 -t ed25519 -~~~~ - -That will create a key named `id_ed25519` in the directory `.ssh/`. If you want to know where that command comes from, read [this article](https://blog.g3rt.nl/upgrade-your-ssh-keys.html). - -Now we need to add that key to Gitea. First open the file `.ssh/id_ed25519.pub` and copy the contents to your clipboard. Now in the Gitea web interface, click on the user menu at the upper right and select "settings". Then across the top you'll see a bunch of tabs. Click the one that reads "SSH / GPG Keys". Click the add key button, give your key a name and paste in the contents of the key. - -Note: depending on how your VPS was set up, you may need to add the `git` user to your sshd config. Open `/etc/ssh/sshd_config` and look for a line that reads something like this: - -~~~~console -AllowUsers myuser myotheruser git -~~~~ - -Add `git` to the list of allowed users so you'll be able to authenticate with the git user over ssh. Now test SSH cloning with this line, substituting your SSH clone url: - -~~~~console -git clone ssh://git@mydomain/giteausername/reponame.git -~~~~ - -Assuming that works then you're all set, Gitea is working and you can create all the repos you need. If you have any problems you can drop a comment in the form below and I'll do my best to help you out. - -If you want to add some other niceties, the Gitea docs have a good guide to [setting up Fail2Ban for Gitea](https://docs.gitea.io/en-us/fail2ban-setup/) and then there's a whole section on [backing up Gitea](https://docs.gitea.io/en-us/backup-and-restore/) that's well worth a read. - -[^1]: You can compile Gitea yourself if you like, there are [instructions on the Gitea site](https://docs.gitea.io/en-us/install-from-source/), but be forewarned its uses quite a bit of RAM to build. diff --git a/tech/set up mysql php.txt b/tech/set up mysql php.txt deleted file mode 100644 index 2856f05..0000000 --- a/tech/set up mysql php.txt +++ /dev/null @@ -1,144 +0,0 @@ - - -Everything you need for wordpress and piwik: - -apt-get install php5-dev libssh2-1-dev libssh2-php php5-geoip libgeoip-dev mysql-server php5-mysql php5-fpm fcgiwrap - -then run: - -sudo mysql_install_db -sudo /usr/bin/mysql_secure_installation - - -mysql -u root -p -CREATE DATABASE local_stats_piwik; -CREATE USER piwiklocalstats@localhost; -SET PASSWORD FOR piwiklocalstats@localhost= PASSWORD(""); -GRANT ALL PRIVILEGES ON local_stats_piwik. TO piwiklocalstats IDENTIFIED BY ''; - -CREATE DATABASE longhandpixels_lhp_wp; -CREATE USER longhandpixels@localhost; -SET PASSWORD FOR longhandpixels@localhost= PASSWORD(""); -GRANT ALL PRIVILEGES ON longhandpixels_lhp_wp. TO longhandpixels IDENTIFIED BY ''; - -FLUSH PRIVILEGES; - -next edit - -sudo vim /etc/php5/fpm/php.ini - -cgi.fix_pathinfo=0 - -open_basedir = '/home/wp-user:/tmp:/home/lxf/git:/var/www/stats.luxagraf.net:/var/www/rss.luxagraf.net:/var/www/rss.longhandpixels.net:/var/www/longhandpixels.net:/var/www/git.luxagraf.net:/var/www/storage.luxagraf.net:/var/www/dev.longhandpixels.net' - -then: - -sudo service php5-fpm restart - -last thing to do for wordpress is create a user for secure updates. reference: https://www.digitalocean.com/community/tutorials/how-to-configure-secure-updates-and-installations-in-wordpress-on-ubuntu - -sudo adduser wp-user -sudo chown -R wp-user:wp-user ~/apps/longhandpixels.net -sudo su - wp-user -ssh-keygen -t rsa -b 4096 # save in /home/wp-user/wp_rsa -(answer blank to everything, including password) -exit -sudo chown wp-user:www-data /home/wp-user/wp_rsa* -sudo chmod 0640 /home/wp-user/wp_rsa* -sudo mkdir /home/wp-user/.ssh -sudo chown wp-user:wp-user /home/wp-user/.ssh/ -sudo chmod 0700 /home/wp-user/.ssh/ -sudo cp /home/wp-user/wp_rsa.pub /home/wp-user/.ssh/authorized_keys -sudo chown wp-user:wp-user /home/wp-user/.ssh/authorized_keys -sudo chmod 0644 /home/wp-user/.ssh/authorized_keys -sudo vim /home/wp-user/.ssh/authorized_keys - -add this to restrict to local connections: -from="127.0.0.1" ssh-rsa... -then - -sudo apt-get update -sudo apt-get install php5-dev libssh2-1-dev libssh2-php -vim apps/longhandpixels.net/wp-config.php - -add these lines: - -define('FTP_PUBKEY','/home/wp-user/wp_rsa.pub'); -define('FTP_PRIKEY','/home/wp-user/wp_rsa'); -define('FTP_USER','wp-user'); -define('FTP_PASS',''); -define('FTP_HOST','127.0.0.1:sshport'); - -restart nginx and it should work. make sure that wp-user in allowed ssh hosts and /home/wp-user/ is in open_basedir in php.ini. - - -## Piwik specific: - -grab piwik: - -wget http://builds.piwik.org/piwik.zip && unzip piwik.zip -mv piwik apps/app.name -sudo chown -R www-data:www-data apps/app.name -mkdir -p /tmp/cache/tracker/ - - -apt-get install php5-gd libfreetype6 # for nice parklines - -How do I install the GeoIP Geo location PECL extension? from http://piwik.org/faq/how-to/#faq_163 - - sudo pecl install geoip - -Finally, add the following to your php.ini file: - - extension=geoip.so - geoip.custom_directory=/path/to/piwik/misc - -Replace /path/to/piwik with the path to your Piwik installation. - -And finally, if you are using the GeoLite City database there is one more thing you need to do. The PECL extension won’t recognize the database if it’s named GeoLiteCity.dat so make sure it is named GeoIPCity.dat. - -in my case : - - cp GeoLiteCity.dat apps/stats.luxagraf.net/misc/GeoIPCity.dat - -sudo chown -R www-data:www-data apps/stats.luxagraf.net/misc/GeoIPCity.dat - -# postgres, postgis python setup - -apt-get install build-essential python python3 python-dev python3-dev python-pip python3-pip python-setuptools -sudo apt-get install postgresql postgresql-server-dev-all -sudo apt-get install binutils libproj-dev gdal-bin postgis postgresql-9.4-postgis-2.1 - - -Stuff for Pillow: -apt-get install libtiff5-dev libjpeg62-turbo-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python-tk - -Install uwsgi: - -PIP_REQUIRE_VIRTUALENV=false -sudo pip3 install uwsgi - -sudo vi /etc/systemd/system/uwsgi.service - - -[Unit] -Description=uWSGI Emperor -After=syslog.target - -[Service] -ExecStart=/usr/local/bin/uwsgi --ini /etc/uwsgi/emperor.ini -Restart=always -KillSignal=SIGQUIT -Type=notify -StandardError=syslog -NotifyAccess=all - -[Install] -WantedBy=multi-user.target - - -sudo mkdir -p /etc/uwsgi/vassals/ && cd /etc/uwsgi/vassals -sudo ln -s ~/apps/luxagraf/config/django.ini /etc/uwsgi/vassals/ -sudo systemctl start uwsgi -sudo systemctl enable uwsgi - |