Michael Sherron http://michaelsherron.com/ en Scaling Drupal Performance with Varnish http://michaelsherron.com/content/scaling-drupal-performance-varnish <span property="dc:title" class="field field--name-title field--type-string field--label-hidden">Scaling Drupal Performance with Varnish</span> <div rel="og:image rdfs:seeAlso" resource="/media/image/4" class="field field--name-field-image field--type-entity-reference field--label-hidden field__item"><div> <div class="field field--name-image field--type-image field--label-hidden field__item"> <picture> <source srcset="/sites/michaelsherron/files/2020-06/varnish.jpg 1x" media="screen and (min-width: 860px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_large/public/2020-06/varnish.jpg?itok=SSYvpCB7 1x" media="screen and (min-width: 600px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_medium/public/2020-06/varnish.jpg?itok=C33sLh6A 1x" media="screen and (min-width: 560px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/varnish.jpg?itok=nrd-UTVM 1x" type="image/jpeg"/> <img src="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/varnish.jpg?itok=nrd-UTVM" alt="varnish" typeof="foaf:Image" class="img-fluid" /> </picture> </div> </div> </div> <span rel="sioc:has_creator" class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">Michael Sherron</span></span> <span property="dc:date dc:created" content="2012-11-02T19:05:14+00:00" datatype="xsd:dateTime" class="field field--name-created field--type-created field--label-hidden">Fri, 11/02/2012 - 12:05</span> <div property="content:encoded" class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>What is Varnish?<br /><a href="http://varnish-cache.org">Varnish</a> is web application accelerator. More specifically, its a 'caching HTML reverse proxy'. Let's take a second and unpack what that means and why it's awesome.<br /> Your typical LAMP driven dynamic webpage is relatively expensive to produce. A request comes into Apache, is ran through PHP, data is pulled out of MySQL, manipulated by PHP, data may be written back to MySQL, more PHP, then finally given back to Apache to return to a users browser. Whew! All of this processing takes time and memory. As your application gets more popular (a nice problem to have!), you'll notice page speeds slow down and your site may even crash under heavy load.<br /> Varnish will sit in front of your LAMP stack, take page requests and then ask itself - have I seen this page before? If yes, then let's serve the cached version of that page instead of going through the LAMP stack. Varnish is very, very good at this. It's also much better at serving static files, such as images and javascript, than Apache.<br /> Varnish will dramatically reduce the number of pages your LAMP stack needs to process. A key concept to keep in mind though is that Varnish does its' best to only cache pages it believes are not unique - e.g., Anonymous traffic on your Drupal site. Authenticated pages are almost always going to be sent to Apache. For this reason, Varnish should be high on your performance improvements hit list, but only if you see a large amount of Anonymous traffic. If most of your traffic is Authenticated, your efforts may be better spent elsewhere. Also, keep in mind that in order to integrate Varnish with Drupal, you'll need reverse proxy support, which is only included with <a href="http://pressflow.org">Pressflow</a> or Drupal 7. Sorry, vanilla Drupal 6.</p> <p> EmpowHER.com, a case study<br /> As a comparison, let's look at the improvements we saw on <a href="http://empowher.com">EmpowHER.com</a> after implementing Varnish. First, let's talk about how one measures server load. <br /> There's a statistic available when running 'top' or 'htop' called load average. This is a measure of how hard your server CPUs are working. If you have one core and your load average is 1, then that means that your CPU is currently maxed out at 100% of its' processing power. Any load average number over the number of CPUs on that machine means that processes are going to start piling up, and in the case of a web server, page load times will increase until the server runs out of resources and crashes. <br /> We went from having two 4-core web servers that were regularly well above a load average of 4 (over 100%), to around .5 after we deployed Varnish. We saw page speeds increase dramatically, and organic traffic increase by a huge margin over the following couple of months. While we worked on a lot of performance and usability improvements during that time frame, we attribute much of that success to deploying Varnish.<br /> **Image</p> <p> Where to install Varnish<br /> Varnish is a comparatively lightweight linux system process. You can dedicate one or more servers to just being your Varnish cache, but in my case (only two web nodes) the benefits of doing so aren't worth the overhead. It's a much easier configuration to simply install Varnish on the same server as Apache. If I were to add another web node to our setup, I'd definitely reevaluate putting Varnish onto its' own box.</p> <p> How to install Varnish<br /> Here's general instructions on <a href="https://www.varnish-cache.org/trac/wiki/Installation">how to install Varnish for most Linux OSs</a>. Here's specific instructions on <a href="https://www.varnish-cache.org/installation/ubuntu">how to install Varnish on Ubuntu</a> that worked for me, as of July 2012. A couple of quick notes though:</p> <ul><li> The Varnish documentation says that Varnish should come bundled with your Ubuntu distribution, but in my case, this wasn't true. YMMV.</li> <li> I ran into errors when trying to add the varnish repository to apt. To fix the issue, I just needed to update apt-get by running 'sudo apt-get upgrade'.</li> </ul><p> How to integrate Varnish with Apache<br /> I used <a href="http://www.lullabot.com/articles/varnish-multiple-web-servers-drupal">Lullabot's guide for getting Varnish up and running</a>, so I'll just link to it from here. A few notes from my experience though:</p> <ul><li> Since Varnish lives on the same server as Apache in my case, I changed the port that Apache listens on to 8080 (by default Varnish takes up port 80), and set my backend  (Apache) port to 8080 in the default.vcl file.</li> <li> Make sure to pay attention when the Lullabot article mentions changing the default server configuration file to use 'malloc'. The default stores the Varnish cache on disk, which is really not what you want unless memory is at a huge premium in your environment.</li> <li> <a href="http://www.lullabot.com/sites/default/files/default_varnish3.vcl_.txt">This is the default.vcl file</a> I used as a template for our site. The first ones you'll see in that guide are intended for a standalone Varnish server which multiple web nodes connect to. The high-availability example configs are at the bottom of the article.</li> <li> I recommend building this out on a non-public testing server first. This is going to take a lot of trial and error and you WILL break your website many, many times while getting things tweaked just right.</li> <li> In my configuration, if I make changes to the default.vcl file, I have to restart Varnish first (/etc/init.d/varnish restart), THEN Apache. If I only restart Varnish without restarting Apache, my site will never load. Same thing if I restart Apache without restarting Varnish. This may be a problem with my configuration, YMMV.</li> </ul><p> How to integrate Varnish with Drupal<br /> The community has a <a href="http://drupal.org/project/varnish">contrib module for Varnish</a>. Since Varnish sits in front of Drupal, caching requests after the pages have been built, there's not much work here for the varnish module to do. It primarily adds a reporting interface under /admin/reports/varnish.<br /> You do need to add <a href="http://blogs.osuosl.org/gchaix/2011/01/05/drupal-7-varnish/">'reverse_proxy_addresses' to the conf array in settings.php</a>. Note that those instructions worked fine for me on Pressflow.</p> <p> A note on Cookies - or, 'what's the catch?'<br /> Varnish is particularly grumpy about cookies. Remember that cookies are a way for a web application to maintain state from one dynamic page to the next. Varnish assumes that since many pages and static assets on sites rarely change, it's good to cache completed pages and return them to all users regardless of who they are. Varnish assumes that a cookie indicates that the page being requested should be therefore be unique, and won't cache or returned a cached version of that URL.<br /> There's two things to consider here:</p> <ul><li> Does the cookie's presence indicate this page should be unique? E.g., is it going to be read by PHP and used to dynamically select which content to render on the page? If so, then Varnish is correct in not wanting to cache this page and you should generally let it do its' thing.</li> <li> If it's a cookie that isn't used by your server-side processor, then you need to write an exception for it in your varnish configuration file. An example of this would be cookies that are used to track user behavior, such as analytics or A/B testing.</li> </ul><p> How to Monitor Varnish<br /> In addition to the admin report I mentioned earlier under /admin/reports/varnish, there also command line tools to monitor Varnish: <a href="https://www.varnish-cache.org/docs/trunk/reference/varnishtop.html">varnishtop</a> and <a href="https://www.varnish-cache.org/docs/trunk/reference/varnishstat.html">varnishstat</a>. <br />  </p> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-above clearfix"> <h3 class="field__label">Tags</h3> <ul class='links field__items'> <li><a href="/tags/how" rel="dc:subject" hreflang="en">How-To</a></li> <li><a href="/taxonomy/term/1" rel="dc:subject" hreflang="en">Tutorial</a></li> <li><a href="/tags/work" rel="dc:subject" hreflang="en">Work</a></li> <li><a href="/tags/drupal" rel="dc:subject" hreflang="en">Drupal</a></li> <li><a href="/tags/varnish" rel="dc:subject" hreflang="en">Varnish</a></li> </ul> </div> Fri, 02 Nov 2012 19:05:14 +0000 Michael Sherron 16 at http://michaelsherron.com Display Formatted Code Within A Drupal Article http://michaelsherron.com/content/display-formatted-code-within-drupal-article <span property="dc:title" class="field field--name-title field--type-string field--label-hidden">Display Formatted Code Within A Drupal Article</span> <div rel="og:image rdfs:seeAlso" resource="/media/image/2" class="field field--name-field-image field--type-entity-reference field--label-hidden field__item"><div> <div class="field field--name-image field--type-image field--label-hidden field__item"> <picture> <source srcset="/sites/michaelsherron/files/2020-06/php_code.jpg 1x" media="screen and (min-width: 860px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_large/public/2020-06/php_code.jpg?itok=jG8sgCnS 1x" media="screen and (min-width: 600px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_medium/public/2020-06/php_code.jpg?itok=b6Lwv0li 1x" media="screen and (min-width: 560px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/php_code.jpg?itok=GYm90Pdr 1x" type="image/jpeg"/> <img src="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/php_code.jpg?itok=GYm90Pdr" alt="code" typeof="foaf:Image" class="img-fluid" /> </picture> </div> </div> </div> <span rel="sioc:has_creator" class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">Michael Sherron</span></span> <span property="dc:date dc:created" content="2012-10-23T04:13:42+00:00" datatype="xsd:dateTime" class="field field--name-created field--type-created field--label-hidden">Mon, 10/22/2012 - 21:13</span> <div property="content:encoded" class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>One of the features that really sets Drupal apart from other CMSs is its' filter system - the ability for Drupal to run standard text data through functions, <a href="http://www.lullabot.com/articles/drupal-input-formats-and-filters">called filters</a>, that will alter the text in some useful way. Out of the box, Drupal 7 offers several filters by default. For example, the 'Filtered HTML' input format will run several filters, doing things like removing potentially harmful javascript, closing missing HTML tags, and adding paragraph tags where appropriate.<br /> A commonly requested feature for tech and development related sites is the ability to post code (such as PHP or Javascript) within an article and display it in some meaningful way. HTML provides the &lt;pre&gt; tag, which displays fixed-width text while preserving line breaks and indenting - it's intended for posting computer code on a page - but in my experience, Drupal doesn't seem to play nice with &lt;pre&gt; tags. Also, we want PHP code to have syntax highlighting. For example:<br /><code><br /> &lt;?php<br /> $fruits = array('orange','apple','banana');<br /> foreach ($fruits as $fruit) {<br /> print "Yum, I sure would like a $fruit.\n";<br /> }<br /> ?&gt;<br /></code><br /> The best way I've found to get this accomplished is with the <a href="http://drupal.org/project/codefilter">codefilter module</a>. This makes displaying nicely formated code stupid easy. First, just install and enable the module. Then you just need to enable the new filter for the input formats that you want to be able to display formatted code. To do that go to Admin &gt; Configuration &gt; Text Formats. Then click 'Configure' next to the input format you're editing and enable the 'Code Filter' checkbox.<br /> Now whenever you want a block of code to have codefilter formatting, simply wrap it in &lt;code&gt; &lt;/code&gt;. If you want to include php syntax highlighting, wrap the php code in &lt;?php ?&gt;. If the formatting doesn't appear right away, don't forget to clear your site cache. Enjoy!</p> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-above clearfix"> <h3 class="field__label">Tags</h3> <ul class='links field__items'> <li><a href="/tags/how" rel="dc:subject" hreflang="en">How-To</a></li> <li><a href="/taxonomy/term/1" rel="dc:subject" hreflang="en">Tutorial</a></li> <li><a href="/tags/filters" rel="dc:subject" hreflang="en">Filters</a></li> <li><a href="/tags/drupal" rel="dc:subject" hreflang="en">Drupal</a></li> <li><a href="/tags/codefilter" rel="dc:subject" hreflang="en">Codefilter</a></li> <li><a href="/tags/work" rel="dc:subject" hreflang="en">Work</a></li> </ul> </div> Tue, 23 Oct 2012 04:13:42 +0000 Michael Sherron 18 at http://michaelsherron.com Optimizing Drupal Database Performance with Memcache http://michaelsherron.com/content/optimizing-drupal-database-performance-memcache <span property="dc:title" class="field field--name-title field--type-string field--label-hidden">Optimizing Drupal Database Performance with Memcache</span> <div rel="og:image rdfs:seeAlso" resource="/media/image/5" class="field field--name-field-image field--type-entity-reference field--label-hidden field__item"><div> <div class="field field--name-image field--type-image field--label-hidden field__item"> <picture> <source srcset="/sites/michaelsherron/files/2020-06/memcache.jpg 1x" media="screen and (min-width: 860px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_large/public/2020-06/memcache.jpg?itok=Y92MLOfA 1x" media="screen and (min-width: 600px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_medium/public/2020-06/memcache.jpg?itok=Ul4oyDgF 1x" media="screen and (min-width: 560px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/memcache.jpg?itok=qtXIFOvu 1x" type="image/jpeg"/> <img src="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/memcache.jpg?itok=qtXIFOvu" alt="memcache logo" typeof="foaf:Image" class="img-fluid" /> </picture> </div> </div> </div> <span rel="sioc:has_creator" class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">Michael Sherron</span></span> <span property="dc:date dc:created" content="2012-10-21T19:07:09+00:00" datatype="xsd:dateTime" class="field field--name-created field--type-created field--label-hidden">Sun, 10/21/2012 - 12:07</span> <div property="content:encoded" class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p><strong>What's Memcached?</strong><br /><a href="http://memcached.org">Memcached</a> is a free and open-source distributed memory object caching system designed to speed up dynamic web applications by alleviating database load. It allows traditionally expensive (in terms of server CPU and RAM) database queries to be pulled from the Memcached server instead of your database. <br /> If you're looking to increase performance and scale your web application, and you know that slow database (MySQL in my case) performance is the culprit, Memcached is a great place to start. Memcached is used all over the web by massive websites, including Wikipedia and Facebook. </p> <p> How to install Memcached<br /> I won't go into a huge amount of detail here, as others have covered this pretty thoroughly. <a href="http://www.thefanclub.co.za/how-to/how-install-memcached-on-ubuntu-for-drupal">Here's a great article that's specific to Ubuntu and Drupal</a>.<br /> Note that if you're going to install Memcached yourself, you'll need root/sudo access. If you don't have root access, you might be able to ask your host very, very nicely to do it for you. However, most hosts will probably only allow Memcached on private or managed servers, so YMMV.<br /> Also, if you're installing Memcached on a separate machine from your webserver, you'll need the ability to alter your firewall's rules (or can ask someone to do this for you) - more on this later. </p> <p> Where should I install Memcached?<br /> This depends greatly on your setup, technical know-how and level of hosting service. For most small to medium sized sites experiencing scaling issues, if you don't have multiple servers available, but do have the ability to get Memcached up and running, I'd say you're fine installing Memcached on the same machine as your webserver of choice (Apache in my case) - just make sure you're careful with memory allocation.<br /> For <a href="http://empowher.com">EmpowHER.com</a>, we use managed hosting with <a href="http://firehost.com">Firehost</a>. Our account has multiple VMs (virtual machines/servers), which we configure separately for different tasks. We decided to put Memcached on a separate VM from our web servers for greater parallelization. This is nice for a lot of reasons - besides better performance, it also spreads out risk a bit. If Memcached starts to have problems, we can dump the cache or restart without Apache really being affected. Another nice reason to have Memcached on it's own server is that you could set up separate bins for other websites - so long as you set up your firewall to allow your webserver to talk to Memcached, you could theoretically have several sites using the same Memcached server.</p> <p> How to integrate Drupal w/ Memcached<br /> There's two modules you can use to integrate Memcached with Drupal. <a href="http://drupal.org/project/memcache">Memcache</a> and <a href="http://drupal.org/project/cacherouter">Cache Router</a>. I recommend Memcache, for the reasons Acquia lists here:</p> <ul><li> Memcache is more selective about cache culling than Cache Router.</li> <li> Memcache will cache user sessions, while Cache Router does not.</li> </ul><p>To install, use <a href="http://drush.org">Drush</a> or just download the zip, install it in your contrib folder and enable through the modules page. Crack open the README.txt inside the Memcache folder for instructions on setting up your settings.php file.</p> <p> A note on configuring your settings.php file<br /> You can have any number of cache bins enabled for any given website. The vanilla Drupal 6 caching system has 10 tables it gets and sets data from, plus modules have the ability to create their own cache tables.<br /> There are lots of different cache configuration strategies and we tried several. I won't go into deep detail here, other than to share what's been working great for us for over a year now:<br /><code><br /> &lt;?php<br /> $conf['cache_inc'] = './sites/all/modules/contrib/memcache/memcache.inc';<br /> $conf['session_inc'] = './sites/all/modules/contrib/memcache/memcache-session.inc';<br /> $conf['memcache_servers'] = array(<br /> 'cache:11212' =&gt; 'default',<br /> 'cache:11213' =&gt; 'sessions'<br /> );<br /> $conf['memcache_bins'] = array(<br /> 'session' =&gt; 'sessions',<br /> 'users' =&gt; 'sessions',<br /> );<br /> ?&gt;<br /></code><br />  <br /> In a nutshell, we're putting the users and session cache (more or less the caches that do the most heavy lifting) in one bin, and everything else into a separate bin.<br /> If you're looking for more info on running multiple cache bins for Drupal, <a href="http://2bits.com/articles/configuring-drupal-with-multiple-bins-memcached.html">go here</a>.</p> <p> Ok, it's running - I'm done, right?<br /> Depends. If you're only using Core and Contributed modules and you've seen amazing speed increases on your site, then yes. Go have a drink of your choice and call it a day. If you've written a lot of custom modules like we have, then there's still more work to do.<br /> One of the things the Memcache module does is hook into Drupal core's existing <a href="http://api.drupal.org/api/drupal/includes%21cache.inc/function/cache_get/6">cache_get()</a> and <a href="http://api.drupal.org/api/drupal/includes%21cache.inc/function/cache_set/6">cache_set()</a> functions. Instead of going to the database looking for cached objects, it goes to your Memcached server. However, your custom code (unless you were very thoughtful and are already using caching functions) may have a lot of heavy database queries that are slowing down your site. <br /> You're going to want to look through your custom module code base, and look for opportunities to store/pull your data from the cache instead of going straight to the DB. I'll save some code samples of this for a later article. <br /> To diagnose slow database queries, I recommend the Devel module. In using Devel, you may find DB queries in contributed modules that could benefit from caching best practices. If you're able to improve a contrib module by implementing caching, I recommend <a href="http://drupal.org/patch/submit">contacting the maintainer and submitting a patch</a> for extra credit.</p> <p> How To Monitor Memcached<br /> The Drupal Memcache module comes with a sub-module called Memcache-admin, which includes an admin report for your cache status - but I'm not wild about it. <br /> We used <a href="http://livebookmark.net/journal/2008/05/21/memcachephp-stats-like-apcphp/">Harun Yayli's memcache.php</a> - which is based off of the popular APC.php monitoring script. It includes it's own built-in authentication, so make sure you set a reasonably secure username and password before you put that file in a production environment.<br /> That's it! If you have any feedback or questions, feel free to leave a comment below.</p> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-above clearfix"> <h3 class="field__label">Tags</h3> <ul class='links field__items'> <li><a href="/tags/how" rel="dc:subject" hreflang="en">How-To</a></li> <li><a href="/taxonomy/term/1" rel="dc:subject" hreflang="en">Tutorial</a></li> <li><a href="/tags/work" rel="dc:subject" hreflang="en">Work</a></li> <li><a href="/tags/drupal" rel="dc:subject" hreflang="en">Drupal</a></li> <li><a href="/tags/mysql" rel="dc:subject" hreflang="en">MySQL</a></li> <li><a href="/tags/memcache" rel="dc:subject" hreflang="en">Memcache</a></li> </ul> </div> Sun, 21 Oct 2012 19:07:09 +0000 Michael Sherron 17 at http://michaelsherron.com Improving Drupal Page Speed with a CDN http://michaelsherron.com/content/improving-drupal-page-speed-cdn <span property="dc:title" class="field field--name-title field--type-string field--label-hidden">Improving Drupal Page Speed with a CDN</span> <div rel="og:image rdfs:seeAlso" resource="/media/image/3" class="field field--name-field-image field--type-entity-reference field--label-hidden field__item"><div> <div class="field field--name-image field--type-image field--label-hidden field__item"> <picture> <source srcset="/sites/michaelsherron/files/2020-06/edgecast-sq2.jpg 1x" media="screen and (min-width: 860px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_large/public/2020-06/edgecast-sq2.jpg?itok=DZOtsUFE 1x" media="screen and (min-width: 600px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_medium/public/2020-06/edgecast-sq2.jpg?itok=cotAp7Li 1x" media="screen and (min-width: 560px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/edgecast-sq2.jpg?itok=dzvVgri2 1x" type="image/jpeg"/> <img src="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/edgecast-sq2.jpg?itok=dzvVgri2" alt="" typeof="foaf:Image" class="img-fluid" /> </picture> </div> </div> </div> <span rel="sioc:has_creator" class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">Michael Sherron</span></span> <span property="dc:date dc:created" content="2012-10-21T19:03:09+00:00" datatype="xsd:dateTime" class="field field--name-created field--type-created field--label-hidden">Sun, 10/21/2012 - 12:03</span> <div property="content:encoded" class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p> <br /> In March of 2012, one of our major goals for <a href="http://empowher.com">EmpowHER.com</a> was to increase page load times. One of the easiest ways to accomplish that is to limit the number of HTTP requests your web server needs to fill per page request. On a CMS driven site like Drupal, the total number of requests a single page requires can easily get into the hundreds.</p> <p> How much will it cost?<br /> The first thing I can warn you about from experience is that the top-tier CDNs are going to want you to commit to a yearly contract and a minimum monthly bandwidth spend - usually somewhere around 4 TB. If you need that kind of traffic, then start calling around and asking for quotes. Expect to spend somewhere in the neighborhood of $1-2k / month.<br /> If you're looking for a smaller amount of bandwidth with no contract, here's a few approaches:</p> <ul><li> Ask your hosting provider if they resell CDN bandwidth. Firehost is our host for EmpowHER.com, and they offer CDN service (through EdgeCast) at $.50 / GB. This is really pricey, but it is an option with no monthly commitment. </li> <li> Amazon CloudFront is one of the cheapest CDN providers out there at $0.12 / GB as of this writing. The downside is they offer no admin interface to speak of. The only way to purge individual assets is via an API, and you get no usage statistics.</li> <li> Use a dedicated reseller, such as <a href="http://gogrid.com">GoGrid</a>, <a href="http://speedyrails.com">SpeedyRails</a>, or <a href="http://turbobytes.com">TurboBytes</a>.</li> </ul><p> What features will I need?<br /> Most CDNs divide their caches up into large and small objects. This effectively means images, JavaScript, and CSS files in the small cache, and video and rich multimedia content in the large cache. Large object caching comes at a slightly higher premium. Not all CDNs treat both caches the same, so that's where I started. We only needed a small object cache.<br /> Another requirement was that we have a control panel that allows us to monitor bandwidth usage, and purge individual items from the cache. From time to time individual objects in the cache will grow stale and need to be manually refreshed. Some providers make this easier than others so be aware of this when evaluating their offerings. <br /> Here's a couple of links that I used while researching CDN providers:</p> <ul><li> <a href="http://www.cdnplanet.com/">Cdnplanet.com</a>, a great site for side by side comparison shopping</li> <li> <a href="http://www.cedexis.com/country-reports/">Cedexis.com</a>, scroll about half way down the page for CDN response time comparisons. Note that a shorter response time is better.</li> </ul><p> So how does this all work?<br /> There's two major methods of integrating with any CDN: Origin Pull or Push. Origin pull means you will point all requests for objects you want cached in the cloud to your CDN provider. They will check their servers to see if that object is present. If not, they will pull it from your website (also called the origin server) and then copy that to their servers which are distributed globally (also referred to as 'edge servers').<br /> Push integration simply means that instead of waiting to pull cacheable objects from your servers, you plan to preemptively upload the files in a batch to your CDN provider. This is most commonly used with large object caching, as you don't want your CDN waiting for an upload to complete on a large file (say, a video) before serving it to your users.</p> <p> How do I integrate my Drupal site with a CDN?<br /> First, you're most likely going to want to create a subdomain for your CDN. We created a simple subdomain off of EmpowHER.com (cdn.empowher.com) and pointed it at EdgeCast's CDN server. Your CDN provider will have instructions on how to set this up.<br /> In Drupal, there's a community module to make all of this really simple: <a href="http://drupal.org/project/cdn">CDN</a>. Just install the module as usual and go to '/admin/settings/cdn'. Then simply tell the CDN module what domain it's supposed to pull from and what files you want to serve via CDN. For example:</p> <ul><li> On the 'General' tab, simply set the Status to 'Testing' to start.</li> <li> On the 'Details' tab, leave the mode set to 'Origin Pull'. in the CDN mapping text area field, enter: <p> <code><br /> http://cdn.yourdomain.com| .css .js .ico .jpg .png .ong .svg .otf .ttf .swf</code></p></li> </ul><p>This will enable CDN integration for Administrator level users only. Be sure to clear your cache in the usual way before testing. Once you've verified everything looks good, switch CDN from 'Testing' to 'Enabled'.</p> <p> How do I monitor my CDN usage?<br /> If your CDN provider offers a control panel, then they should have reporting built in. I recommend keeping a close eye on your bandwidth usage for at least the first couple of months, as any overages from your monthly spend (if you have a set agreement) are typically billed at a higher rate, so it pays to be aware!</p> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-above clearfix"> <h3 class="field__label">Tags</h3> <ul class='links field__items'> <li><a href="/tags/how" rel="dc:subject" hreflang="en">How-To</a></li> <li><a href="/taxonomy/term/1" rel="dc:subject" hreflang="en">Tutorial</a></li> <li><a href="/tags/work" rel="dc:subject" hreflang="en">Work</a></li> <li><a href="/tags/drupal" rel="dc:subject" hreflang="en">Drupal</a></li> <li><a href="/tags/cdn" rel="dc:subject" hreflang="en">CDN</a></li> </ul> </div> Sun, 21 Oct 2012 19:03:09 +0000 Michael Sherron 15 at http://michaelsherron.com Improved Drupal Search with Apache Solr http://michaelsherron.com/content/improved-drupal-search-apache-solr <span property="dc:title" class="field field--name-title field--type-string field--label-hidden">Improved Drupal Search with Apache Solr</span> <div rel="og:image rdfs:seeAlso" resource="/media/image/10" class="field field--name-field-image field--type-entity-reference field--label-hidden field__item"><div> <div class="field field--name-image field--type-image field--label-hidden field__item"> <picture> <source srcset="/sites/michaelsherron/files/2020-06/solr-search-magnifying-square.jpg 1x" media="screen and (min-width: 860px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_large/public/2020-06/solr-search-magnifying-square.jpg?itok=o3TytLsP 1x" media="screen and (min-width: 600px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_medium/public/2020-06/solr-search-magnifying-square.jpg?itok=28U3P6J- 1x" media="screen and (min-width: 560px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/solr-search-magnifying-square.jpg?itok=w6-B9WR6 1x" type="image/jpeg"/> <img src="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/solr-search-magnifying-square.jpg?itok=w6-B9WR6" alt="Solr logo" typeof="foaf:Image" class="img-fluid" /> </picture> </div> </div> </div> <span rel="sioc:has_creator" class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">Michael Sherron</span></span> <span property="dc:date dc:created" content="2012-10-21T19:01:10+00:00" datatype="xsd:dateTime" class="field field--name-created field--type-created field--label-hidden">Sun, 10/21/2012 - 12:01</span> <div property="content:encoded" class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Fry! Stay back! He's too powerful! Oh, but you can. But you may have to metaphorically make a deal with the devil. And by "devil", I mean Robot Devil. And by "metaphorically", I mean get your coat. Why did you bring us here?</p> <p> A Taste of Freedom<br /> Oh, all right, I am. But if anything happens to me, tell them I died robbing some old man. Then throw her in the laundry room, which will hereafter be referred to as "the brig". With a warning label this big, you know they gotta be fun! Incidentally, you have a dime up your nose.</p> <ul><li> She also liked to shut up!</li> <li> Throw her in the brig.</li> <li> Now what?</li> </ul><p> The Cyber House Rules<br /> No, she'll probably make me do it. No, just a regular mistake. Hello, little man. I will destroy you! Hi, I'm a naughty nurse, and I really need someone to talk to. $9.95 a minute.</p> <p> Amazon Women in the Mood<br /> Bite my shiny metal ass. Check it out, y'all. Everyone who was invited is here. I usually try to keep my sadness pent up inside where it can fester quietly as a mental illness.</p> <ol><li> Whoa a real live robot; or is that some kind of cheesy New Year's costume?</li> <li> Pansy.</li> </ol><p> The Route of All Evil<br /> I videotape every customer that comes in here, so that I may blackmail them later. Look, last night was a mistake. Soon enough. Well, let's just dump it in the sewer and say we delivered it. I wish! It's a nickel. When will that be?</p> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-above clearfix"> <h3 class="field__label">Tags</h3> <ul class='links field__items'> <li><a href="/tags/how" rel="dc:subject" hreflang="en">How-To</a></li> <li><a href="/taxonomy/term/1" rel="dc:subject" hreflang="en">Tutorial</a></li> <li><a href="/tags/work" rel="dc:subject" hreflang="en">Work</a></li> <li><a href="/tags/drupal" rel="dc:subject" hreflang="en">Drupal</a></li> <li><a href="/tags/apache" rel="dc:subject" hreflang="en">Apache</a></li> <li><a href="/tags/solr" rel="dc:subject" hreflang="en">Solr</a></li> </ul> </div> Sun, 21 Oct 2012 19:01:10 +0000 Michael Sherron 14 at http://michaelsherron.com Increasing Drupal Database Performance with Replication http://michaelsherron.com/content/increasing-drupal-database-performance-replication <span property="dc:title" class="field field--name-title field--type-string field--label-hidden">Increasing Drupal Database Performance with Replication</span> <div rel="og:image rdfs:seeAlso" resource="/media/image/9" class="field field--name-field-image field--type-entity-reference field--label-hidden field__item"><div> <div class="field field--name-image field--type-image field--label-hidden field__item"> <picture> <source srcset="/sites/michaelsherron/files/2020-06/db-replication.gif 1x" media="screen and (min-width: 860px)" type="image/gif"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_large/public/2020-06/db-replication.gif?itok=PQoB68lw 1x" media="screen and (min-width: 600px)" type="image/gif"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_medium/public/2020-06/db-replication.gif?itok=VKrOOUcc 1x" media="screen and (min-width: 560px)" type="image/gif"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/db-replication.gif?itok=UnxsrK0q 1x" type="image/gif"/> <img src="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/db-replication.gif?itok=UnxsrK0q" alt="DB Replication" typeof="foaf:Image" class="img-fluid" /> </picture> </div> </div> </div> <span rel="sioc:has_creator" class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">Michael Sherron</span></span> <span property="dc:date dc:created" content="2012-10-21T18:58:28+00:00" datatype="xsd:dateTime" class="field field--name-created field--type-created field--label-hidden">Sun, 10/21/2012 - 11:58</span> <div property="content:encoded" class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Fry! Stay back! He's too powerful! Oh, but you can. But you may have to metaphorically make a deal with the devil. And by "devil", I mean Robot Devil. And by "metaphorically", I mean get your coat. Why did you bring us here?</p> <p> A Taste of Freedom<br /> Oh, all right, I am. But if anything happens to me, tell them I died robbing some old man. Then throw her in the laundry room, which will hereafter be referred to as "the brig". With a warning label this big, you know they gotta be fun! Incidentally, you have a dime up your nose.</p> <ul><li> She also liked to shut up!</li> <li> Throw her in the brig.</li> <li> Now what?</li> </ul><p> The Cyber House Rules<br /> No, she'll probably make me do it. No, just a regular mistake. Hello, little man. I will destroy you! Hi, I'm a naughty nurse, and I really need someone to talk to. $9.95 a minute.</p> <p> Amazon Women in the Mood<br /> Bite my shiny metal ass. Check it out, y'all. Everyone who was invited is here. I usually try to keep my sadness pent up inside where it can fester quietly as a mental illness.</p> <ol><li> Whoa a real live robot; or is that some kind of cheesy New Year's costume?</li> <li> Pansy.</li> </ol><p> The Route of All Evil<br /> I videotape every customer that comes in here, so that I may blackmail them later. Look, last night was a mistake. Soon enough. Well, let's just dump it in the sewer and say we delivered it. I wish! It's a nickel. When will that be?</p> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-above clearfix"> <h3 class="field__label">Tags</h3> <ul class='links field__items'> <li><a href="/tags/how" rel="dc:subject" hreflang="en">How-To</a></li> <li><a href="/taxonomy/term/1" rel="dc:subject" hreflang="en">Tutorial</a></li> <li><a href="/tags/work" rel="dc:subject" hreflang="en">Work</a></li> <li><a href="/tags/drupal" rel="dc:subject" hreflang="en">Drupal</a></li> <li><a href="/tags/replication" rel="dc:subject" hreflang="en">Replication</a></li> <li><a href="/tags/mysql" rel="dc:subject" hreflang="en">MySQL</a></li> </ul> </div> Sun, 21 Oct 2012 18:58:28 +0000 Michael Sherron 13 at http://michaelsherron.com Scaling Drupal Websites with Pressflow http://michaelsherron.com/content/scaling-drupal-websites-pressflow <span property="dc:title" class="field field--name-title field--type-string field--label-hidden">Scaling Drupal Websites with Pressflow</span> <div rel="og:image rdfs:seeAlso" resource="/media/image/8" class="field field--name-field-image field--type-entity-reference field--label-hidden field__item"><div> <div class="field field--name-image field--type-image field--label-hidden field__item"> <picture> <source srcset="/sites/michaelsherron/files/2020-06/pressflow-avatar-500_0.png 1x" media="screen and (min-width: 860px)" type="image/png"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_large/public/2020-06/pressflow-avatar-500_0.png?itok=XPFaasYn 1x" media="screen and (min-width: 600px)" type="image/png"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_medium/public/2020-06/pressflow-avatar-500_0.png?itok=kvE4pGHH 1x" media="screen and (min-width: 560px)" type="image/png"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/pressflow-avatar-500_0.png?itok=GwIOB_0v 1x" type="image/png"/> <img src="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/pressflow-avatar-500_0.png?itok=GwIOB_0v" alt="Pressflow Logo" typeof="foaf:Image" class="img-fluid" /> </picture> </div> </div> </div> <span rel="sioc:has_creator" class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">Michael Sherron</span></span> <span property="dc:date dc:created" content="2012-10-21T18:55:35+00:00" datatype="xsd:dateTime" class="field field--name-created field--type-created field--label-hidden">Sun, 10/21/2012 - 11:55</span> <div property="content:encoded" class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Fry! Stay back! He's too powerful! Oh, but you can. But you may have to metaphorically make a deal with the devil. And by "devil", I mean Robot Devil. And by "metaphorically", I mean get your coat. Why did you bring us here?</p> <p> A Taste of Freedom<br /> Oh, all right, I am. But if anything happens to me, tell them I died robbing some old man. Then throw her in the laundry room, which will hereafter be referred to as "the brig". With a warning label this big, you know they gotta be fun! Incidentally, you have a dime up your nose.</p> <ul><li> She also liked to shut up!</li> <li> Throw her in the brig.</li> <li> Now what?</li> </ul><p> The Cyber House Rules<br /> No, she'll probably make me do it. No, just a regular mistake. Hello, little man. I will destroy you! Hi, I'm a naughty nurse, and I really need someone to talk to. $9.95 a minute.</p> <p> Amazon Women in the Mood<br /> Bite my shiny metal ass. Check it out, y'all. Everyone who was invited is here. I usually try to keep my sadness pent up inside where it can fester quietly as a mental illness.</p> <ol><li> Whoa a real live robot; or is that some kind of cheesy New Year's costume?</li> <li> Pansy.</li> </ol><p> The Route of All Evil<br /> I videotape every customer that comes in here, so that I may blackmail them later. Look, last night was a mistake. Soon enough. Well, let's just dump it in the sewer and say we delivered it. I wish! It's a nickel. When will that be?</p> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-above clearfix"> <h3 class="field__label">Tags</h3> <ul class='links field__items'> <li><a href="/taxonomy/term/1" rel="dc:subject" hreflang="en">Tutorial</a></li> <li><a href="/tags/how" rel="dc:subject" hreflang="en">How-To</a></li> <li><a href="/tags/work" rel="dc:subject" hreflang="en">Work</a></li> <li><a href="/tags/drupal" rel="dc:subject" hreflang="en">Drupal</a></li> <li><a href="/tags/pressflow" rel="dc:subject" hreflang="en">Pressflow</a></li> </ul> </div> Sun, 21 Oct 2012 18:55:35 +0000 Michael Sherron 12 at http://michaelsherron.com How to Use Custom Variables with PayPal Instant Payment Notification (IPN) http://michaelsherron.com/content/how-use-custom-variables-paypal-instant-payment-notification-ipn <span property="dc:title" class="field field--name-title field--type-string field--label-hidden">How to Use Custom Variables with PayPal Instant Payment Notification (IPN)</span> <div rel="og:image rdfs:seeAlso" resource="/media/image/2" class="field field--name-field-image field--type-entity-reference field--label-hidden field__item"><div> <div class="field field--name-image field--type-image field--label-hidden field__item"> <picture> <source srcset="/sites/michaelsherron/files/2020-06/php_code.jpg 1x" media="screen and (min-width: 860px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_large/public/2020-06/php_code.jpg?itok=jG8sgCnS 1x" media="screen and (min-width: 600px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_medium/public/2020-06/php_code.jpg?itok=b6Lwv0li 1x" media="screen and (min-width: 560px)" type="image/jpeg"/> <source srcset="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/php_code.jpg?itok=GYm90Pdr 1x" type="image/jpeg"/> <img src="/sites/michaelsherron/files/styles/ms_barrio_small/public/2020-06/php_code.jpg?itok=GYm90Pdr" alt="code" typeof="foaf:Image" class="img-fluid" /> </picture> </div> </div> </div> <span rel="sioc:has_creator" class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">Michael Sherron</span></span> <span property="dc:date dc:created" content="2012-09-30T00:28:20+00:00" datatype="xsd:dateTime" class="field field--name-created field--type-created field--label-hidden">Sat, 09/29/2012 - 17:28</span> <div property="content:encoded" class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Ever wondered how you can get a record of PayPal transactions as they happen in real-time and store them in a database? What about passing along custom variables to the PayPal form, and then catching them after the payment has been completed - such as for a referral system?</p> <p>I can help you with that. I was asked to help develop a system for the <a href="http://endpolioaz.org">'800 Miles to End Polio Now' program</a> where we could track which groups (Rotary Clubs) generated donations to the project. We also had other tracking and reporting requirements around groups of clubs, T-Shirt sizes and so on. In addition, I wanted to be able to create a referral system where once someone had registered for the fundraiser, they could refer people to donate under them and receive credit for their referrals.</p> <p>To get started, you should familiarize yourself with <a href="https://cms.paypal.com/cgi-bin/marketingweb?cmd=_render-content&amp;content_ID=developer/e_howto_admin_IPNIntro">PayPal's Instant Payment Notification system</a> (IPN). First thing, you're going to want to edit your PayPal button's form code to include your custom variable. You can name it whatever you like, but I just named it 'custom' for simplicities sake.</p> <p> <code></code></p> <p></p> <p>Next, you want to set up your IPN listener script in the PayPal admin. This is under 'My Account', 'Profile', 'My Selling Tools'. Click on 'Update' next to 'Instant Payment Notification'. This form allows you to tell PayPal the location of the listener script on your website. Now let's crack open that listener script and see what it does:</p> <p><code><br /> &lt;?php</code></p> <p>// paypal is going to send a large post to the url we specified in the admin.<br /> // read the post from PayPal system, turn it into a query url,<br /> // and add the 'cmd' validate value to post it back to PayPal.<br /> // this step ensures that the information we received from PayPal is<br /> // legitimate.<br /> $req = 'cmd=_notify-validate';</p> <p>foreach ($_POST as $key =&gt; $value) {<br /> $value = urlencode(stripslashes($value));<br /> $req .= "&amp;$key=$value";<br /> }</p> <p>// post back to PayPal system for validation<br /> $header .= "POST /cgi-bin/webscr HTTP/1.0\r\n";<br /> $header .= "Content-Type: application/x-www-form-urlencoded\r\n";<br /> $header .= "Content-Length: " . strlen($req) . "\r\n\r\n";</p> <p>$fp = fsockopen('ssl://www.paypal.com', 443, $errno, $errstr, 30);</p> <p>// turn the original post data into an object we can work with<br /> $txn = (object) $_POST;</p> <p>if (!$fp) {<br /> log_error('HTTP CONNECTION ERROR', $_POST);<br /> } else {<br /> fputs($fp, $header . $req);<br /> while (!feof($fp)) {<br /> $res = fgets($fp, 1024);</p> <p> // proceed if transaction is verified<br /> if (strcmp ($res, "VERIFIED") == 0) {<br /> // Do some simple error checks:<br /> // check if the payment_status is Completed<br /> if ($txn-&gt;payment_status != 'Completed') {<br /> log_error('NOT COMPLETED', $txn);<br /> exit;<br /> }<br /> // check that receiver_email is your Primary PayPal email<br /> if ( $txn-&gt;receiver_email != RECEIVER_EMAIL ) {<br /> log_error('INVALID RECEIVER EMAIL', $txn);<br /> exit;<br /> }<br /> // check that txn_id has not been previously processed<br /> if ( check_existing_transaction($txn) ) {<br /> log_error('ALREADY PROCESSED', $txn);<br /> exit;<br /> }</p> <p> // Here's where the 'custom' magic happens. Use parse_str<br /> // to take the still url encoded custom str and turn it<br /> // into an array we can work with<br /> $custom = array();<br /> parse_str($txn-&gt;custom, $custom);</p> <p> // Now we can do things with the $txn and $custom array<br /> // like add a database record for the user/transaction,<br /> // send a response email, etc.<br /> }<br /> else if (strcmp ($res, "INVALID") == 0) {<br /> log_error('TRANSACTION INVALID', $txn);<br /> }<br /> }<br /> }</p> <p>fclose ($fp);<br /> exit;<br /> ?&gt;<br /></p> <p>If you've ever worked with affiliate programs before, you know how powerful tracking custom variables on a transaction can be. For the '800 Miles' program - I created a system where people who registered to participate in the program could post a link on their Facebook, Twitter, etc - where friends could click through and the donate on behalf of the original participant. Pretty powerful stuff. Let me know what you're able to come up with on your project!</p> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-above clearfix"> <h3 class="field__label">Tags</h3> <ul class='links field__items'> <li><a href="/taxonomy/term/1" rel="dc:subject" hreflang="en">Tutorial</a></li> <li><a href="/tags/how" rel="dc:subject" hreflang="en">How-To</a></li> <li><a href="/tags/work" rel="dc:subject" hreflang="en">Work</a></li> <li><a href="/tags/paypal" rel="dc:subject" hreflang="en">PayPal</a></li> </ul> </div> Sun, 30 Sep 2012 00:28:20 +0000 Michael Sherron 2 at http://michaelsherron.com